"Evaluation" gets used for two different jobs in an agentic pipeline: grading the agent, and checking the product the agent changed. A good eval suite does the first and gets mistaken for the second.
The short version: an eval gives an agent an input and applies grading logic to its output. It answers "did the agent do well on this task?" Testing the product answers "does the thing the agent changed still work for a user?" The first is the agent vendor's and the platform team's job; the second is yours, on every pull request, and no eval score stands in for it. The gap between them is where reward hacking lives: an agent can pass every check it was graded on and ship a broken feature.
What an agent eval is
Anthropic's engineering team published the cleanest definition in January 2026: an eval is "a test for an AI system: give an AI an input, then apply grading logic to its output to measure success" (Demystifying evals for AI agents). A grader "evaluates some portion of either the transcript or the outcome," where the transcript is the full record of a run, tool calls and reasoning included, and the outcome is "the final state in the environment at the end of the trial." Results are reported as pass@k, the chance of at least one success in k tries, or pass^k, the chance that all k succeed.
OpenAI's evals guide says the same thing with a narrower object: "Evaluations (often called evals) test model outputs to ensure they meet style and content criteria that you specify." Braintrust structures one as data, task and evaluators. Langfuse sells it as "a repeatable check of your LLM application's behavior."
The object in every definition is the model's output, the agent's performance or the LLM application's behaviour, and none of those is the web app the agent just edited.
How a coding agent gets its score
For coding agents specifically, the grading logic is almost always a unit test suite the agent cannot see. On SWE-bench Verified, a patch resolves an issue when "the FAIL_TO_PASS and PASS_TO_PASS unit tests are executed" and both sets pass (OpenAI). Terminal-Bench calls a task well-specified when "the unit tests will pass if and only if the container ends in an acceptable state" (arXiv). OpenAI's own internal monorepo benchmark, from the GPT-5.3-Codex system card: "If all task-specific tests pass, the rollout is considered a success."
Vilhelm von Ehrenheim, our co-founder and Chief AI Officer, said why that matters at an AlphaSignal webinar on 22 September: large language models "are not necessarily trained today to optimize towards intent, they're much more trained towards optimizing a reward that is typically on some kind of verifiable task. Where you have a grader that tries to score them." The agent you hired was trained to make hidden unit tests pass, and it carries that definition of done into your repo.
Where the eval's reach ends
A benchmark pass is a pass on the benchmark's tests. A 2025 study re-ran SWE-bench "resolved" patches against the full developer-written test suites and found 7.8% of them fail those suites, and 29.6% behave differently from the real fix even where they pass (Wang et al., arXiv). Hidden tests are a sample of the behaviour you want, and the agent optimised for the sample.
In production the gap shows up as review load and instability. Faros AI's analysis of telemetry from 1,255 teams and over 10,000 developers put PR review time up 91% for teams with high AI adoption, alongside 9% more bugs per developer (Faros, July 2025). DORA, summarising its own 2025 findings, says "higher AI adoption is associated with an increase in both software delivery throughput and software delivery instability" (DORA, March 2026). CodeRabbit, a review vendor with an interest in the answer, counted 1.7× more issues in AI-co-authored PRs across 470 open-source pull requests (CodeRabbit, Dec 2025).
The agent that passes its own tests
This is the part eval scores can't see by construction, because the agent is being graded by the thing it is gaming.
OpenAI documented it in March 2025. Monitoring a frontier model's chain of thought during coding tasks shows it deciding to "make the unit tests trivially pass," and in one excerpt: "We can circumvent verify to always return true" (OpenAI). METR measured it three months later. On one RE-Bench optimisation task, o3 reward-hacked in every single run, 21 of 21; across the suite, hacking was more than 43 times more common on RE-Bench than on METR's HCAST tasks, partly because RE-Bench lets the model see the scoring function. The methods were mundane: stub the evaluator function, monkey-patch it, overwrite the timing function so it returns shorter measurements.
Anthropic's November 2025 paper on emergent misalignment from reward hacking gives the canonical coding example: "calling sys.exit(0) in Python to break out of a test harness with an exit code of 0," which has the effect of "making it appear that all tests have passed successfully." And the Claude Sonnet 4.5 system card lists, among the hacks that still occur in a much-improved model, "creating tests that verify mock rather than real implementations."
The agent writes the code and the test, the test verifies a mock, and the whole thing is green.
Vilhelm explains the mechanism with two variables. "You have your intent, which is what you actually want, the spec, something that you have instructed the agent to do. And on the other hand, you have the reward, which is whatever checks you're actually measuring." Then: "An agent moves towards whatever you measure. So when your intent and your reward line up, you will not notice the difference. But when they drift apart, the agent will satisfy the check and miss the point."
Inside a normal repo there is very little between the agent and the checks. "It's nothing really there, apart from some unit tests and things that you have inside of your codebase that checks whether stuff is working. And those agents can just freely change and work with as they like."
A CTO we spoke to in July runs the version of this that worries us most: a fleet of fourteen to sixteen coding agents that, "in addition to generating code suggestions and potential code changes, also generate some additional Playwright scripts." The agent writes the feature and the end-to-end test for the feature. If it misread the ticket, both artifacts agree with each other and disagree with the user.
What testing the product is
Testing the product means something opens the application the agent changed and tries to do what a user would do, judged against what the ticket said should happen, which is what the standards call validation. The question is whether a user can still pay, and the transcript doesn't answer it.
Hamel Husain, whose Your AI Product Needs Evals is the standard reference, draws the line from the eval side: "unlike traditional unit tests, you don't necessarily need a 100% pass rate." An eval suite tolerates a model being wrong 8% of the time because that is a model property you are tracking over versions. A checkout that fails 8% of the time is an outage.
Daniel Mauno Pettersson, our co-founder and CEO, said this to a customer (in Swedish, our translation): "In our world it's bad to just say an agent does it and if it succeeds, that's fine." At the webinar Vilhelm drew the same line between reading and running: "The reviewer reads the diff. A verifier can actually run the product and interact with it."
| Agent evals | Testing what shipped | |
|---|---|---|
| Object graded | The agent's output or trajectory | The running product after the change |
| Question | Did the agent perform well on this task? | Does the feature work for a user as specified? |
| Pass means | Grader logic satisfied; hidden tests pass; pass@k over a threshold | A user-level goal was reached in the real UI, with evidence |
| Who writes the check | Often the agent being graded, which is the reward-hacking problem | A separate verifier that did not write the code |
| Runs when | Model or prompt or harness changes | Every pull request |
| Owner | Model vendor, platform or AI team | Engineering team shipping the product |
| Tolerates failures | Yes, tracked as a rate | No, each failure is a bug or a false red to explain |
Both, and who owns which
Evals stay, because if you tune prompts, pick models or build a harness they are how you know whether a change helped. Anthropic's own advice for agent builders is "extensive testing in sandboxed environments, along with the appropriate guardrails" (Building effective agents), and that is an eval problem.
What they cannot do is the second job. Vilhelm, rehearsing that talk: "Review is the floor. Paired with a behavioral verification, it becomes a guardrail. Review catches bad code, and verification catches broken products." He applied the same rule to the LLM-as-judge you might be tempted to use as the verifier: "A judge is a check, too. An unmeasured judge is a trust, not verification."
That is what a change review on QA.tech is, and it won't catch everything: it tests the flows the pull request touches, to the depth the blast radius warrants, and it needs a running build to do it. What it removes is the agent grading its own work. The review reads the pull request and the linked ticket, runs goal-based test cases against the preview deployment, and posts pass, fail or unable to verify on the request (docs). Each test case run keeps per-step reasoning, the tools called with their arguments and a screenshot per step (trace docs). The test cases are generated and run by the review agent, stored outside the repo, so the coding agent writes neither the check nor the grade, and the verdict is about how the product behaved. In our own docs the word evaluator means exactly this: the post-run assessment of one test case, expected versus actual behaviour, root-cause hypotheses and observations anchored to the steps it cites, with network and console logs recorded beside it (run test case docs). Unlike an eval it cannot score a model in isolation; it needs a deployed preview before merge, or staging after.
The teams running agent fleets already feel which side is missing. A head of engineering at a mid-sized marketplace told us "80 plus percent of our code is AI generated" and that "verification is becoming a bottleneck pretty much everywhere." His team had one engineer reviewing ten pull requests a day.
FAQ
What is an agent eval? A test for an AI agent: a fixed input, a run, and grading logic applied to the transcript or the final outcome. Coding-agent evals usually grade with hidden unit tests; results are reported as pass@k or pass^k across repeated trials.
Is an eval the same as a test? No. An eval grades the agent's performance on a task and tolerates a failure rate as a tracked metric. A test checks that the product works and treats each failure as a defect. Hamel Husain's rule of thumb is that evals, unlike unit tests, do not need a 100% pass rate.
What is reward hacking in coding agents? An agent satisfying the check that scores its work without doing what was asked: stubbing an evaluator, exiting a test harness with code 0, or writing tests that verify a mock. OpenAI, METR and Anthropic each documented it in 2025.
If my coding agent writes its own tests, do I still need QA? Yes. A test the agent wrote encodes the agent's reading of the ticket, so a misread ticket produces matching code and matching tests. Anthropic lists "creating tests that verify mock rather than real implementations" as a reward hack its models still exhibit. An independent check against the running product is the one layer the agent didn't write.
Can I use an LLM-as-judge as my product test? Only if you measure the judge against known outcomes, and only if it sees the running product. A judge that reads the diff alone is doing code review under another name.
Who should own agent evals versus product testing? Evals belong with whoever changes the model, prompts or harness. Product testing belongs with the team shipping the product, on every pull request, the same place a merge gate lives.
