# Evals in CI for LLM agents, without a golden answer

> How to build an eval set from production traffic, score an LLM agent without a golden answer, and fail a merge in CI when it regresses.

Source: https://meshin.co/blog/evals-in-ci-for-llm-agents
Updated: 2026-10-07

---

An eval suite that only runs on a laptop protects nobody. The same regression a developer catches by eye on a good day slips through on a bad one. A model provider's silent update slips through every time. Putting [an eval suite](/glossary/evals) in CI turns a habit into a gate. The pull request cannot merge until the numbers clear, whoever is watching.

## How do you build an eval set from production traffic?

Pull the first cases from what the agent has actually handled, not from what a spec said it should handle. Real traces carry the phrasing, the edge cases, and the odd multi-step requests. A person writing test data from imagination never thinks to include them. A support agent's eval set should read like a week of its own transcripts, because that is exactly what it is.

Sample across a period long enough that one good or bad week does not decide the result. A month of traffic spans weekdays, weekends, and spikes the business already expects, which is a sturdier base than one convenient week.

Mix the set on purpose. Include three kinds of case: routine ones that should always pass, a handful known to be hard, and a few with no single acceptable answer. Most agent tasks in production fall into that last kind. Store the set next to the code it tests, in a format a reviewer can read in a pull request. Change it through the same review a code change gets.

Revisit the set on a fixed schedule rather than whenever someone remembers. Retire a case once the product has genuinely outgrown it. Add one the day a new failure reaches production. That keeps the set tracking how people actually use the agent.

## How do you score an agent when there is no golden answer?

Most agent tasks do not reduce to one correct string, so do not force them into that shape. A support reply, a summary, or a multi-step tool sequence can be right in more than one way. A scorer built for exact match will reject good answers as often as it catches bad ones. Score in layers instead, cheapest and most reliable checks first.

Start with deterministic checks on anything mechanical. Check whether the agent called the right tool, with a valid argument set, in a shape that downstream code can actually parse. These checks are cheap, and they never disagree with themselves. In most agentic systems, they catch a large share of regressions before a judge ever runs.

OWASP's list of risks in large language model applications names excessive agency as one of the standard failure modes worth testing for. Excessive agency means an agent taking an action its task never asked for. Write a case for every tool the agent can call. Each case should check that the tool stays quiet when not needed, and that it fires correctly when needed.

For the answer's content, add a reference-free judge. Look at the question and whatever the agent actually retrieved. Then check whether every claim in the answer traces back to a document it had in hand. This catches a fluent, confident answer that is not supported by anything the system looked at. A plain accuracy score misses that failure mode entirely.

Where no rubric fits cleanly, fall back to a comparison against the last known-good version. Show a judge both outputs, blind to which one is new, and ask it to pick a winner. A judge finds picking a winner easier than defending an absolute score out of some invented scale. A win rate against a fixed baseline is also a steadier number to watch than a score with no anchor.

NIST's AI RMF frames this as a selection problem under its Measure function. Which method fits depends on what is actually being measured. A team documents why it chose a metric, and which ones it rejected. That team ends up with a sturdier suite than one that picked whatever was easiest to compute.

## What happens when the judge itself is flaky?

A judge model can score the same input differently on two runs, especially at a higher temperature. That alone will make a pipeline look unstable, even when the agent under test has not changed. Set the judge's own temperature as low as the provider allows. Treat any remaining run-to-run disagreement as a property of the judge to manage.

Calibrate the judge on a fixed schedule against a small set of cases a human has already scored. The number that matters is agreement between the judge and the human reviewers. A judge that is consistent but consistently wrong is worse than one that is honestly uncertain. Consistent wrongness hides the problem instead of showing it.

Let the judge decline a case it cannot confidently assess. A skipped case should fall out of the pass-rate calculation, rather than count as either a pass or a fail. A case the judge skips often is telling you the rubric does not cover that situation yet.

When one specific case flips between pass and fail on identical re-runs, treat that case as broken, not the whole gate. Quarantine it. Keep it running so you can see it, stop it from blocking a merge, and put a name against fixing its rubric or its sampling. Only then should it rejoin the set that actually gates a build.

## What should the pipeline check, and what does a failing check do to the build?

Not every check should carry the same weight. A wrong tool call and a slow response are both real problems, but only one of them should stop a release. The table below is the shape of a working gate. It shows what each row measures, how it gets a number, and what that number does to the build when it fails.

| What to score                            | How to score it                                                                                          | What a failing score does to the build                                                          |
| ---------------------------------------- | -------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| Task success on labelled cases           | Exact match or a deterministic rule against a known-good answer.                                         | Blocks the merge immediately, with no override.                                                 |
| Task success with no golden answer       | A rubric-based judge scores the answer against stated criteria, not against a reference.                 | Blocks the merge once the pass rate drops below the floor set for that task.                    |
| Faithfulness to retrieved context        | A reference-free judge checks whether each claim traces back to a document the agent actually retrieved. | Blocks the merge; an unsupported claim counts as a defect, not a style note.                    |
| Preference against the last good version | Pairwise comparison: the judge sees both outputs, blind to which is new, and picks a winner.             | Blocks the merge if the new version loses on more cases than the agreed margin allows.          |
| Tool-call correctness                    | Deterministic check that the agent called the right tool with a valid, schema-checked argument set.      | Blocks the merge; a malformed call fails loudly instead of degrading quietly.                   |
| Judge-human agreement                    | Re-run the rubric against a human-labelled sample on a fixed schedule and compare agreement.             | Does not block a single pull request; a drop pages the eval owner to fix the rubric.            |
| Cost per completed run                   | Sum of model and tool cost across the eval set, divided by cases completed.                              | Blocks the merge once the run exceeds the ceiling set in the pipeline config.                   |
| Response time under load                 | Wall-clock time per case, measured in the same job as the scores.                                        | Warns on the pull request; blocks only if it breaches the budget agreed with the product owner. |

## Where does the cost ceiling sit?

Put a cost ceiling in the same job as the quality score. Do not put it on a separate dashboard someone checks once a week. Agent systems fail commercially long before they fail technically. Consider a tool call that retries a few times on a bad response: harmless in a demo, ruinous once real traffic hits it.

Measure cost per completed run, across the whole eval set, the way you measure the pass rate. A change that makes the agent slower or more expensive without a real gain is a regression, even if every other score stays green. A [90-Day Production Sprint](/services/90-day-production-sprint) puts this gate in place before a workflow takes its first real user. That is a cheaper time to find the problem than after the first invoice.

## What does the pipeline do when a check fails?

A failing check blocks the merge by default. The pull request gets a summary next to the diff that caused it. That summary lists which cases failed, what the agent returned, and what the check expected. Nobody reconstructs the failure from a log.

Not every failure deserves the same response. A wrong tool call or an unsupported claim should stop the release outright. A latency or cost warning close to its budget, or a drop in judge-human agreement, can page an owner instead, without stopping every other change in the queue. Keep that distinction explicit in the pipeline config, not in someone's memory of which checks are actually load-bearing.

Only a named reviewer, not the author of the change, can override a hard block. That override has to write itself into the same log the eval results live in. An override that leaves no trace is indistinguishable from no gate at all. It gets used that way the next time someone is in a hurry.