A practical guide to writing evals that catch real regressions

An eval suite that always passes is not a safety net, it is decoration. If your scores have not moved in two months, either your product is perfect or your evals are not measuring anything a user would notice.

Start from complaints, not from theory

The best eval corpus is your support queue. Take the last fifty escalations, find the traces behind them, and label what went wrong in plain language. You will usually find three or four recurring failure shapes — a hallucinated policy, a missing citation, a tone that reads as dismissive. Those become your rubrics.

Resist the urge to write a general-purpose quality score. A single number that averages groundedness, tone and completeness will move slowly and tell you nothing about what to fix.

Judge models need judging too

If you use an LLM as a grader, hold out a set of human-labelled examples and measure the grader against them. A judge that agrees with your team seventy percent of the time is a coin flip with extra steps, and you will spend weeks chasing regressions that are really grader noise.

Re-check that agreement every time you change the judge prompt or the underlying model. It drifts more than people expect.