Evals in CI for LLM agents, without a golden answer

An eval suite that only runs on a laptop protects nobody. The same regression a developer catches by eye on a good day slips through on a bad one. A model provider’s silent update slips through every time. Putting an eval suite in CI turns a habit into a gate. The pull request cannot merge until the numbers clear, whoever is watching.

How do you build an eval set from production traffic?

Pull the first cases from what the agent has actually handled, not from what a spec said it should handle. Real traces carry the phrasing, the edge cases, and the odd multi-step requests. A person writing test data from imagination never thinks to include them. A support agent’s eval set should read like a week of its own transcripts, because that is exactly what it is.

Sample across a period long enough that one good or bad week does not decide the result. A month of traffic spans weekdays, weekends, and spikes the business already expects, which is a sturdier base than one convenient week.

Mix the set on purpose. Include three kinds of case: routine ones that should always pass, a handful known to be hard, and a few with no single acceptable answer. Most agent tasks in production fall into that last kind. Store the set next to the code it tests, in a format a reviewer can read in a pull request. Change it through the same review a code change gets.

Revisit the set on a fixed schedule rather than whenever someone remembers. Retire a case once the product has genuinely outgrown it. Add one the day a new failure reaches production. That keeps the set tracking how people actually use the agent.

How do you score an agent when there is no golden answer?

Most agent tasks do not reduce to one correct string, so do not force them into that shape. A support reply, a summary, or a multi-step tool sequence can be right in more than one way. A scorer built for exact match will reject good answers as often as it catches bad ones. Score in layers instead, cheapest and most reliable checks first.

Start with deterministic checks on anything mechanical. Check whether the agent called the right tool, with a valid argument set, in a shape that downstream code can actually parse. These checks are cheap, and they never disagree with themselves. In most agentic systems, they catch a large share of regressions before a judge ever runs.

OWASP’s list of risks in large language model applications names excessive agency as one of the standard failure modes worth testing for. Excessive agency means an agent taking an action its task never asked for. Write a case for every tool the agent can call. Each case should check that the tool stays quiet when not needed, and that it fires correctly when needed.

For the answer’s content, add a reference-free judge. Look at the question and whatever the agent actually retrieved. Then check whether every claim in the answer traces back to a document it had in hand. This catches a fluent, confident answer that is not supported by anything the system looked at. A plain accuracy score misses that failure mode entirely.

Where no rubric fits cleanly, fall back to a comparison against the last known-good version. Show a judge both outputs, blind to which one is new, and ask it to pick a winner. A judge finds picking a winner easier than defending an absolute score out of some invented scale. A win rate against a fixed baseline is also a steadier number to watch than a score with no anchor.

NIST’s AI RMF frames this as a selection problem under its Measure function. Which method fits depends on what is actually being measured. A team documents why it chose a metric, and which ones it rejected. That team ends up with a sturdier suite than one that picked whatever was easiest to compute.

What happens when the judge itself is flaky?

A judge model can score the same input differently on two runs, especially at a higher temperature. That alone will make a pipeline look unstable, even when the agent under test has not changed. Set the judge’s own temperature as low as the provider allows. Treat any remaining run-to-run disagreement as a property of the judge to manage.

Calibrate the judge on a fixed schedule against a small set of cases a human has already scored. The number that matters is agreement between the judge and the human reviewers. A judge that is consistent but consistently wrong is worse than one that is honestly uncertain. Consistent wrongness hides the problem instead of showing it.

Let the judge decline a case it cannot confidently assess. A skipped case should fall out of the pass-rate calculation, rather than count as either a pass or a fail. A case the judge skips often is telling you the rubric does not cover that situation yet.

When one specific case flips between pass and fail on identical re-runs, treat that case as broken, not the whole gate. Quarantine it. Keep it running so you can see it, stop it from blocking a merge, and put a name against fixing its rubric or its sampling. Only then should it rejoin the set that actually gates a build.

What should the pipeline check, and what does a failing check do to the build?

Not every check should carry the same weight. A wrong tool call and a slow response are both real problems, but only one of them should stop a release. The table below is the shape of a working gate. It shows what each row measures, how it gets a number, and what that number does to the build when it fails.

Where does the cost ceiling sit?

Put a cost ceiling in the same job as the quality score. Do not put it on a separate dashboard someone checks once a week. Agent systems fail commercially long before they fail technically. Consider a tool call that retries a few times on a bad response: harmless in a demo, ruinous once real traffic hits it.

Measure cost per completed run, across the whole eval set, the way you measure the pass rate. A change that makes the agent slower or more expensive without a real gain is a regression, even if every other score stays green. A 90-Day Production Sprint puts this gate in place before a workflow takes its first real user. That is a cheaper time to find the problem than after the first invoice.

What does the pipeline do when a check fails?

A failing check blocks the merge by default. The pull request gets a summary next to the diff that caused it. That summary lists which cases failed, what the agent returned, and what the check expected. Nobody reconstructs the failure from a log.

Not every failure deserves the same response. A wrong tool call or an unsupported claim should stop the release outright. A latency or cost warning close to its budget, or a drop in judge-human agreement, can page an owner instead, without stopping every other change in the queue. Keep that distinction explicit in the pipeline config, not in someone’s memory of which checks are actually load-bearing.

Only a named reviewer, not the author of the change, can override a hard block. That override has to write itself into the same log the eval results live in. An override that leaves no trace is indistinguishable from no gate at all. It gets used that way the next time someone is in a hurry.