A coding agent can only fix what a failing test tells it. It needs the scope of the failure (which change, which user flow), evidence it can read (the failing step, a screenshot, the network and console logs) and a way to rerun the check. It also needs a verdict that separates a broken product from a check that couldn't run, and a hint on whether retrying makes sense. And it needs all of that before merge, somewhere it already looks: the pull request, or a tool it can call.
On 28 October debuff published a comparison of 12 JavaScript CLIs, read from the side of the agent running them. The agent gets two things back, the exit code and stdout. In debuff's results, ten of the twelve tools reject an option they don't have, only two give that mistake a different exit code from a real failure, and Jest and Vitest both exit 1 either way. Nothing in the output says whether trying again would help. It's a vendor's bench, one platform and one run, and the author says so.
debuff spells out the consequence: "An agent that reads a typo'd flag as failing tests goes looking for a bug in code that's fine."
We've made the short version of this argument before, in the behaviour sensor a coding-agent harness is missing and in the four layers of verifying coding agent work. Both stop at one sentence on what the agent should get back. This piece goes field by field: what does a test failure have to contain for a coding agent to act on it?
The agent acts on whatever you hand it
A person who sees a confusing red build asks a colleague, or opens the logs and squints. An agent does whatever its context suggests next. And a lot of the time nobody is watching. LangChain, describing the five building blocks of its internal "company harness", says a large share of its agent runs are scheduled or triggered in the background by events, with no user kicking them off. So the failure report is the whole briefing. When it's thin, three things go wrong.
The agent learns to retry. If rerunning a failing test turns it green now and then, Addy Osmani writes in his essay on agentic verification, the agent will settle on rerunning as its strategy.
It chases bugs that aren't there. Hand a coding agent a false red and it comes back with a commit, a review round and a change to code that was fine.
It loops. The Claude Code team hit this in their own code review. "A person can wave off a nit in seconds. An agent fixes it, which means another commit and another round of review." One of their PRs went through more than fifty rounds of review and fixes overnight before its author stopped it, and their rule now is that after three rounds the reviewer asks the author to step back and reconsider the approach.
The quiet version of the same problem is a pass that means nothing. Armin Catovic's team watched an agent run the same successful SQL query 113 and 254 times in two runs, because the tool returned a 20-row preview the agent didn't recognise as one. His conclusion fits test results too: "A successful tool call is not necessarily a useful tool call." A green run that never touched the changed code is exactly that kind of success.
Feedback the agent can still pick up
Timing first, because it decides whether anything below gets read at all.
An agent working on a pull request can read a review on that PR. It can call a tool and read the answer. It can't pick up a nightly regression run on main that finishes after its session has ended. By then the agent that wrote the change is gone, its context is gone, and the failure turns into a ticket for a human.
That's the slow path for QA. Tickets get filed and developers pick them up when they can, long after the change. Put the check inside the loop the agent is already working in and you've shifted left about as far as testing can go.
The feedback has to land on the PR before a human opens it, in a form the agent can use without help.
The fields, one by one
Scope: what changed, and which flow broke
First the agent needs to know whether the failure is its job. Tie the result to the diff and to a user flow: checkout, applying a discount code, on this PR's preview, in a flow the change to cart/pricing.ts can reach. A failure in a flow the PR never touched may be real. It's still someone else's, and an agent handed it is likely to try fixing it anyway. If your system can tell a failure that was already there from one the change introduced, put that in the report.
Evidence: the step, the screenshot, the logs
"'Timeout after 30s' tells you nothing," as Osmani puts it. What helps is "Screenshots, tracing, and an explanation of which step failed," which "tells the agent what went wrong." The minimum is the step where the run broke, expected against actual in plain words, a screenshot of that moment, and the network and console output around it. A 422 from the discount endpoint can be the whole diagnosis.
Be generous with the detail on what happened against what was expected. If the agent can also reach your own traces and logs, say through an MCP server on your stack, it can pull in the backend side of the failure and line it up with the screenshot and the API call that failed. With all of that in hand, the agent has far less to guess.
There's a ceiling too. Thoughtworks' explainer on context rot names test output as one cause: "A test run, for example, might produce hundreds of lines when only a failing assertion and a few stack-trace lines are relevant." The same piece warns that "aggressive filtering can remove useful evidence." Layers solve both. A short summary goes in the report. Everything else stays fetchable: screenshots by step, logs filtered by URL or status code, the full trace when the agent asks.
Reproduction
The agent has to be able to run the failing check again, on its own, against its fix. That takes a stable test ID, the environment the test ran in (which preview URL, which build) and the data it used. Rerunning only the failed tests is the most useful action in the whole loop. History helps as well. A test that fails the same way twice is telling the agent something. One that flips between runs is telling it something else.
A verdict with more than two values
A pass/fail pair has nowhere to put a check that couldn't do its job, the point we made in passing tests are not proof. The agent needs to tell apart at least four cases:
- The product is broken. Fix the code.
- The check couldn't verify. The preview was down, the login expired, the test never reached the change. Fix nothing and flag it.
- The test, or the testing agent, got it wrong. Rerun or fix the test, and leave the code alone.
- The command was wrong. debuff's mistyped flag. Fix the invocation.
Only the first should send an agent into your source code. Ben Balter wrote about documentation last month that "markup is a contract; formatting is a costume". Verdicts work the same way. A category the agent can branch on is a contract. A red icon in a comment is a costume.
Whether a retry makes sense
"Give errors a stable code, and say whether retrying can help," debuff asks of tool authors. Where you can, do the retrying before the agent ever sees the result. Rerun a suspected timing failure once or twice, then report flaky as flaky, in its own section. Where you can't, say what kind of failure it was, so retrying never becomes the agent's default. Catovic's advice from the SQL loop belongs in the harness as well: "Detect repeated equivalent calls, enforce sensible execution budgets."
Where the result lands
The pull request comes first because every coding agent already knows what a PR is. Any of them can read a review, a failing check and a comment, so the PR is the default channel. What it should carry as proof is covered in proof-carrying pull requests. The second channel is a tool the agent can call. Over MCP, a coding agent can ask QA.tech how its tests went and why one failed, pull the evidence, and rerun the failed tests once it has pushed a fix. The PR holds the verdict. The tool serves the evidence on demand.
The checklist
| Field | Why the agent needs it | Bad | Good |
|---|---|---|---|
| Scope | Decides whether the failure is its job | "E2E suite failed (3/214)" | "Checkout › apply discount code fails on this PR's preview; the flow is reachable from the change to cart/pricing.ts" |
| Expected vs actual | Turns a red light into a target | "AssertionError: expected true to be false" | "Expected the total to drop from $50.00 to $45.00 after code SAVE10. It stayed at $50.00." |
| Step and screenshot | Shows where in the flow it broke | "Error at checkout.spec.ts:88" | "Step 6 of 9, after clicking Apply. Link to the screenshot of that step." |
| Network and console | Can hold the cause the assertion only hints at | The full HAR file pasted into the comment | "POST /api/cart/discount returned 422. No console errors." |
| Reproduction | Lets it confirm the fix | "Run the suite again" | "Rerun this test ID against this preview URL, failed tests only" |
| Verdict category | Decides whether to touch the code | A red X | "Fail: product behaviour" or "Unable to verify: login expired" |
| Retry advice | Keeps retrying from becoming a strategy | Nothing | "Failed twice the same way. Not timing." |
| Location | Decides whether it ever sees the result | A nightly email to the QA channel | A required check and a review on the PR, with details fetchable over MCP or an API |
The example values are illustrative.
Can a scripted suite give an agent all this?
A good part of it, yes. Playwright's default config records a trace on the first retry of a failed test, and its trace viewer shows the network, errors and console for each step. Version 1.65 added a failures reporter that prints failures and successful retries as they occur. In debuff's bench, 6 of the 7 CLIs with a JSON mode print parseable JSON when a run fails. A wrapper script can also give a usage error its own exit code, the way ESLint and webpack already do.
Write assertion messages in product terms, tag tests by user flow and post the summary to the PR, and a scripted suite gets a lot easier for an agent to use. Basis, describing how it made its monorepo ergonomic for agents, runs a verifier sub-agent that "runs diff-scoped tests and pre-commit hooks, then reports pass/fail with actionable failure details." If your suite is in good shape, start there.
What a wrapper can't change is that a scripted test reports on the script. When the flow changes, the failure says the locator timed out. That's true about the test and silent about the product, so the agent has to work out which of the two is wrong. If it can't tell, it may well edit the test until it goes green.
There's also the question of who wrote the test. The agent that wrote the code tends to write it too, from the same reading of the ticket. TDD still matters for agents, since an agent needs feedback on its own work as it goes. The verdict on the change needs a check on top of that, one the agent loop has no control over.
That's the argument for context-based testing on the pull request. A testing agent that knows the product, reads the diff and the PR description, and runs the affected flows in the preview reports in the product's terms: what the user tried, what should have happened, what happened. The coding agent hears what its change did to the product, not which selector timed out.
None of this retires the scripted suite. Sign-up and payment still deserve a short scheduled run that does the same thing every time. What moves is the check on each pull request, which becomes dynamic testing on the diff, reporting to the coding agent in the terms above.
Worked example: a PR review verdict, then MCP
QA.tech is an AI testing platform that tests pull requests on their preview deployments and returns the result to the PR, and to coding agents over MCP. It doesn't fill every row of the checklist, and the gaps are at the end of this section. Here is how the rest maps onto it, in an illustrative scenario.
A coding agent opens a PR that changes how discount codes are validated. The preview deploys. The review reads the diff and the PR description, maps the change to the checkout flow and sets a risk level from the blast radius, how central that flow is and what's already covered. It picks the existing tests that cover that flow, writes new ones only where there's a gap, and runs them in the preview.
On the PR, the result is a native GitHub review with one of three verdicts: pass, fail or unable to verify. With it come what was tested, a results summary, a table of the tests and a link through to the full evidence. The review only names a bug once a test has reproduced it. Make the QA.tech / PR Review check required in branch protection and the review becomes your merge gate. That gate is GitHub only. On GitLab the review posts a note and doesn't fail the pipeline.
The evidence isn't in the comment. Screenshots, the step trace, logs and, where captured, the recording live in the run, one click away. For a person that's a link. For a coding agent it's a handful of MCP calls:
get_runwith the failed test cases.get_run_test_casefor the goal, the error classification and the evaluator's assessment: expected against actual behaviour, root-cause hypotheses and observations anchored to steps.get_run_test_case_screenshotsfor the steps the assessment cites, returned as images.get_run_test_case_logsfor network and console output, filtered by URL pattern or status code.get_test_case_history, to see whether this test has failed like this before.rerun_runwithfailedOnlyonce the fix is pushed, or withskipAgentCacheso every step is worked out fresh.
In the example, the assessment says the total should have dropped to $45.00 after the code was applied and stayed at $50.00. The cited screenshot is step 6, after clicking Apply. The network log shows the discount endpoint returning 422. The agent has scope, evidence and a direction before anyone opens the PR.
Now the gaps. There's no separate reproduction script: reproducing a failure means rerunning it, with the steps and trace alongside. Runs aren't identical every time either. They're repeatable on unchanged pages. The finer failure categories (product bug, outdated test, test data, timing, agent mistake, environment, credentials) come from the regression review agent on test plan runs. That agent reruns a suspect test at most twice and reports flaky tests in their own section. PR reviews give the three-value verdict. Web testing is Chromium only, too.
MCP access is included from the Starter plan. Setup for Claude Code, Cursor and Codex is in the MCP docs, along with every tool the agent can call. For other options, see our comparison of QA tools for coding agents, or Playwright MCP if you'd rather have the agent drive the browser itself.
FAQ
What should a test failure include for a coding agent?
It needs the flow and change it relates to, and expected against actual behaviour in plain words. Add the step where it broke with a screenshot, the network and console output around that step, a way to rerun just that test, a verdict category and whether a retry could help. Keep the summary short and make the rest of the evidence fetchable on request.
Why isn't an exit code enough for an AI agent?
An exit code is one bit, and test runners use the same code for very different problems. In debuff's comparison of 12 JavaScript CLIs, only 2 gave a mistyped option a different exit code from a real failure, and none said whether retrying would help. An agent that can't tell a usage error from a failing test can end up hunting for a bug in code that works.
How do you stop a coding agent from retrying flaky tests?
Do the retries before the agent sees the result, report flaky tests separately from failures, and label each failure with its kind so a retry is only attempted where it can help. Put an execution budget on the loop and cap the number of review rounds per pull request.
Should a coding agent get test results from the pull request or over MCP?
Both. The pull request is the channel every coding agent already reads, so the verdict and a short summary belong there as a review or a required check. A tool interface such as MCP or an API lets the agent pull screenshots, logs and traces only when it needs them, which keeps its context small.
Should test output for a coding agent be JSON?
Structure matters more than format. A JSON report lets the agent read the verdict, the failing step and the test ID without parsing prose. That only works if it lands on stdout or at a path the agent knows, and if setup errors come back in the same shape. Most of all, each failure needs a stable kind the agent can branch on: product broken, couldn't verify, test wrong or command wrong.
How much test output should go into an agent's context?
Less than you'd think. Long logs crowd out the part that matters and degrade how well the model uses the rest of its context. Give the failing step, expected against actual and the one or two log lines that explain it. Leave the full trace available behind a call.
What's the difference between a failed test and "unable to verify"?
A failure means the check ran and the product didn't behave as expected. "Unable to verify" means the check couldn't establish the result at all, for example because the tests ran but none of them reached the changed code, or the app couldn't be reached to try. An agent should fix code after the first and leave the code alone after the second.