TL;DR An eval is a test suite for a system that does not give the same answer twice. You collect real inputs, define what a good answer looks like, and score the model against them automatically. Without evals you are shipping on vibes. With them, “it feels better” becomes a number you can defend, and a regression becomes something you catch before your users do.
Ordinary software is testable because it is deterministic: the same input gives the same output, so a test can assert equality and be done. A language model breaks that assumption. Ask it the same question twice and you can get two different answers, both reasonable, one subtly wrong. So the question every serious AI project runs into is not “does it work,” it is “how often, and how do you know.” Evals are the answer to that question, and they are the single clearest line between someone who has shipped an AI feature and someone who has only demoed one.
What an eval actually is
An eval has three parts. A set of inputs, the kind of requests your system will really see. A definition of success for each one, whether that is an exact answer, a set of facts that must appear, or a rubric. And a grader that scores the model's output against that definition and gives you a number. Run all three together and you get a score, say 84% on 200 cases, that you can track over time. That is the whole idea. It is a test suite, adapted for a system whose output you cannot predict exactly.
Building the set
The eval set is the asset, not the score. Start small and real: twenty to fifty inputs pulled from actual usage beat a thousand you invented, because invented cases test the system you imagined instead of the one people use. Cover the common path, then deliberately add the hard cases, the ambiguous request, the question with no good answer, the input designed to trip the system up. Every time production surprises you with a failure, that failure becomes a new row in the set. Over time the eval stops being a snapshot and becomes a memory of every way the system has ever broken.
Grading answers you cannot predict
The hard part is scoring output that has no single correct string. You have a ladder of options, and you climb it only as far as you need. Cheapest and most reliable is a programmatic check: exact match, a regex, valid JSON against a schema, or “does the answer contain these required facts.” Push as much as you can down to this rung, because it is deterministic and free. When the quality is genuinely subjective, tone, helpfulness, faithfulness to a source, you reach for an LLM as the judge: a second model scoring the first against a written rubric.
LLM-as-judge is powerful and it is not free. A judge has its own biases, it can favour longer answers or its own phrasing, so you validate it against human ratings before you trust it, and you keep the rubric specific enough that two people would grade the same way. Structured output helps here: have the judge return { "score": 1-5, "reason": "..." } so the result is machine-readable and the reason is auditable. Tools like OpenAI's evals framework exist precisely to wire this loop together.
Evals in the deployment loop
An eval earns its keep the moment you change something. A new system prompt, a cheaper model, a tweak to retrieval: each one is a bet that quality held or improved, and the eval is how you settle the bet before it reaches a customer. This is regression testing for AI. Run the set on every meaningful change, watch the number, and refuse to ship the ones that drop it. The most valuable version tracks a metric the business already cares about, resolution rate, escalations avoided, time saved, so the score is not an abstraction but a proxy for the thing you were actually hired to move.
Why this is the job
Nobody deploys a system they cannot measure. A prototype gets excitement in a room; an eval is what gets a pilot signed off into production, because it turns “trust me” into evidence a stakeholder can check. When you are deployed alongside a customer, the eval set is often the most useful thing you build, more than any single prompt, because it is the shared definition of “good” that everyone can point at.
So when a model change makes something feel better, I have learned not to trust the feeling. I add the case that worried me to the set, run it, and read the number. If I cannot measure the improvement, I have to assume I imagined it, and more often than I would like, I did.