A green test suite tells you that the assertions someone wrote down are still true. That is a real signal and worth having. It is not the same as evidence that the running product works, because nobody ever wrote an assertion for the defect that is about to ship. The gap has always existed. What changed in 2026 is how much code is arriving per week and how little of it any one person has read, which makes a green check a thinner thing to merge on than it used to be.
Through October 2026 several teams who are not talking to each other reached for the same fix: watch the running product, on top of the tests. Meta built agents that read the live state of a simulator. A plugin in Cursor's GitHub org, pstack, makes driving the real app the condition for calling work done. Cloudflare's security skill hands every finding to a second agent that tries to disprove it.
None of those is a study. Three teams reaching for the same shape of answer is not evidence that the shape works.
What the measured version actually says
The number people quote comes from METR. In March 2026 they had maintainers from scikit-learn, Sphinx and pytest blind-review 296 AI-written patches that had already passed the SWE-bench automated grader. The headline: "roughly half of test-passing SWE-bench Verified PRs written by mid-2024 to mid/late-2025 agents would not be merged". Maintainer merge decisions came in "about 24 percentage points lower" than the automated grader's scores, once normalised against that human baseline. Unnormalised the gap is 34.9 points.
The control group matters, though. "about 68% of golden patches are indeed merged by our maintainers." Those are the human-written reference patches, and roughly a third of them got rejected too, after passing the same tests. So a meaningful slice of that gap is reviewer judgement, house style, and the friction of someone else's patch landing in your repo. It is not all broken software. METR is also measuring SWE-bench, a benchmark of library issues, not a product with a signup flow and a billing page.
The second measured claim is SWE-ABS (arXiv:2603.00520, posted 28 February 2026). It re-runs the top thirty agents' patches against strengthened suites: program slicing to add tests for untested code, then plausible-but-wrong patches generated to find what the tests still miss. It strengthens suites for 50.2% of instances and rejects 19.71% of previously passing patches; the top agent falls from 78.80% to 62.20%. "However, we show that this performance is inflated." Their finding: "one in five 'solved' patches from the top-30 agents are semantically incorrect". One in five is a lot. It is also not half, and it was found by writing harder tests, which is worth keeping in mind for the next section.
So the measured version is narrower than the version you hear at conferences. Passing tests are an imperfect proxy for correct, by somewhere between one in five and one in two depending on what you measure, and a chunk of the larger number is taste rather than bugs.
The obvious objection: just write better tests
Simon Ritter, deputy CTO at Azul, quoted by LeadDev on the METR results, reads the rejections the other way round. "Where there are rejections, it would indicate that there may not be sufficient tests." If a patch passes your suite and is still wrong, your suite was incomplete. Strengthen it.
He's right about a large part of it, and SWE-ABS is the evidence. They found their one-in-five by generating stronger tests. Mutation testing, coverage gaps, missing boundary cases: all of that is real, all of it is fixable with effort, and a team that does it will catch defects a team that doesn't will ship. If your suite is thin, the first and cheapest move is to make it less thin. Nothing below replaces that.
One thing cuts against him before we get to the limits of tests. METR's own control says rejection and insufficient tests are not the same thing: about a third of the human-written golden patches were turned down too, by the same maintainers, after passing the same tests. Those maintainers did not have thin suites. They had opinions.
Here is the slice better tests cannot reach. A test asserts something a person predicted. That is the whole mechanism. You imagine a way the code could be wrong, you encode it, and the test holds the line against that specific failure forever after. Which means the set of things your suite can catch is bounded by the set of things somebody thought of.
Look at what actually went wrong in JK Gunnink's early-access review of Gemini 4 Argon, published 20 October 2026. He was building three things through Antigravity, one of them a redesigned UI for RocketFlag, his own feature-flagging SaaS. "Despite 895 unit tests passing on the first run, the live UI had major visual defects". The refactor "silently removed navigation and edit controls, broke accessible menu names with unhidden badges". He has a whole section headed "Gap Between Passing Tests and Browser Fidelity".
Now ask what test would have caught the removed navigation control. Someone would have had to assert that the nav item still exists, not because they suspected a refactor would delete it, but as a standing guard against a thing nobody does on purpose. You can write that assertion. You cannot write all of them, for every control on every screen, forever. This is one developer's n=1 account, not a study.
Gunnink's own read is that the model "consistently exhibits visual blind spots and concurrency edge cases without live execution feedback." His response was process, not more tests: "I mandated an adversarial self-review before opening any PRs", and "integrating closed-loop verification (such as automated visual harnesses, runtime logs, or structured self-review)".
Better tests fix the predicted failures. Observation of the running product is what you have left for the unpredicted ones.
What the running product shows that a suite does not
Meta's engineering blog described this on 15 October 2026, writing up Forum, "a standalone Groups app that a small team built from scratch". Their framing is "treating an AI agent as a participant in the engineering process rather than a faster way to type", and in their loop the agent plans, implements, builds, tests, corrects and repeats. "Agents check their own work before an engineer sees it."
The part worth stealing is what the agents were allowed to look at. The agents "read the live view hierarchy from the simulator", and the team built "repeatable stress scenarios that mimic real user behavior". Not the test results. The live view hierarchy is what is actually on screen, in a running build, right now. That is a different input from a pass/fail boolean. Meta is a vendor describing its own practice here and publishes no before/after defect count. They also keep a human in it: "Every shipped outcome still carries an engineer's sign-off", and the architecture "compiles prototype code out of the production build entirely".
The same instinct shows up in pstack, a plugin by Lauren Tan that lives in Cursor's own GitHub org. Its pitch is "if you want to go fast, go deep first." Its skill-creation guide sets a bar for what counts as done: "Every serious project needs a scripted way to drive the real app and prove behavior: launch it, exercise a feature the way a user would, and capture evidence." The generated skill is told to exercise a feature the way a user would rather than through internal setters or test-only endpoints, and to capture the action and the resulting state, not just the final screen. Evidence means "Screenshots, terminal transcripts, response bodies, logs, exit codes, DB state". And the behaviour that does the most work: before handing the generated skill over, pstack runs it once from start to finish. The verify-and-ship guide is specific about UI. It requires that "a UI change walks the changed flow". Its "Prove It Works" principle "checks the real artifact."
Cloudflare's security-audit-skill (47.5k stars as of 26 October 2026) adds the independence requirement. "The agent that checks a finding is never the agent that found it." The design "orchestrates isolated agents through reconnaissance, coverage-led hunting, candidate validation", and it will "give every unique candidate to a fresh verifier that tries to disprove it". Security findings are not UI defects, but the structural point transfers. The thing that produced the work should not be the only thing that signs it off. The merge-gate piece traces the same rule through Uber, Sphinx and Gittifacts.
What the three share is a conclusion: read the artifact, not the report about the artifact.
Customers describe the same hole without using any of that language. A senior engineering lead at a collaboration-software company named the hole on a call: "we have different tools to evaluate, to review from the AI to ensure the code is good. But we are not sure from the end user perspective, is it good enough." Later in the same conversation: "But that doesn't actually say the whole user journey works on the production. It says the functionality is working fine." Two different statements about two different objects, and his tooling only covered one of them.
What we built instead
Three things it will not do: non-Chromium browsers, native iOS and Android in the same run, or hand you Playwright or code to run somewhere else. If you need those, this isn't the layer for you.
On a pull request, QA.tech, an AI testing platform, puts an agent in front of your running product. It is not the agent that wrote the code. It has already crawled the app in a pass somebody started and holds a knowledge graph of what exists, and it reads the diff to decide what is worth exercising this time. Then it opens the app and does it, the way a user would.
The verdict is pass, fail, or "unable to verify". The third exists because a binary forces a guess: a guess that lands on "pass" is worse than no answer. Behind each verdict sit screenshots, network and console logs, and a per-step trace of what the agent did and why. On an AlphaSignal webinar in September, co-founder Vilhelm von Ehrenheim compared the two outputs: "some classical regression tests are just kind of green or red, essentially. It's very hard to kind of dig out what the reasons are for that fail."
Where it fits against what you already run: unit and integration tests stay where they are, which is why 100% test coverage isn't the goal is not an argument against coverage so much as against treating it as the finish line. Code review stays too. An engineering director at a small B2B shop described the ceiling on that one: "Yeah, I mean, we do code reviews, but obviously that's not going to catch major issues or things that are underlying issues that you can't see by a code review, you know." The running-product check is a fourth thing, after static analysis, the suite, and review. We wrote up how those four layers stack in how to verify coding agent work, and who gates the merge now covers what happens when the fourth layer becomes the gate.
One caution. An agent driving a live app produces a different flakiness profile than a scripted suite, and the useful distinction between the two kinds of flaky still applies: an environment problem is not a product problem, and a verdict that conflates them will train your team to ignore the whole channel.
The volume problem underneath all of this
On 15 October 2026 the pseudonymous efp.asia blog described agents working "inside a dense ten-year-old codebase": "they produce plausible code that takes longer to check than it would have taken to write". The same post concedes that "The productivity story isn't clean, and I'd be cheating if I pretended otherwise." Fair enough. But plausible-and-unread is what makes a green check load-bearing in a way it never was before. When a human wrote the patch, the suite was one signal and the author's understanding was another. When an agent writes it and the diff runs 900 lines, the suite is doing that work alone.
That is the practitioner version of the argument, and it is wider than anything METR or SWE-ABS measured. Both looked at benchmark patches under a grader, not at a shipped product. We have not seen anyone measure it on a shipped product.
What to do about it this week
Pick one flow that would embarrass you if it broke. Checkout, signup, whatever yours is. Then check whether anything in your pipeline opens the running app and walks that flow end to end on a pull request. Not a test of a function the flow calls. The flow.
If the answer is no, you have found the gap, and the fix does not have to be a purchase. A script that launches the app, walks the path and saves screenshots is a day's work.
If the answer is yes, check the next thing. Does it produce evidence you can look at, or just a colour? If it is a colour, you will stop reading it by March.
FAQ
Do passing tests mean my code works?
They mean the behaviours someone chose to assert still hold. That is a genuine regression signal. It says nothing about the behaviours nobody asserted, which is where most shipped defects live. METR found that "roughly half of test-passing SWE-bench Verified PRs written by mid-2024 to mid/late-2025 agents would not be merged" by maintainers reviewing them blind.
Isn't the answer just to write better tests?
Partly, yes. SWE-ABS found its one-in-five figure by re-running patches against coverage-driven and mutation-driven strengthened suites, so stronger tests do find real defects. The limit is structural: a test checks a prediction. Gunnink's missing navigation controls and broken accessible menu names were not things anyone had predicted.
What is "closed-loop verification"?
Gunnink uses it to cover "automated visual harnesses, runtime logs, or structured self-review", anything that feeds the result of running the code back into the loop before the PR opens. pstack phrases the same requirement as driving the real app and capturing evidence of the action and the resulting state. The common element is observation of behaviour rather than a report about it.
Does this mean AI-written code is worse than human-written code?
The METR data doesn't support that reading on its own. About 68% of the human-written golden patches were merged by the same maintainers, so roughly a third of human control patches that passed tests were also turned down. Reviewer judgement and house style account for a real share of both numbers.
Is there a study showing agentic QA reduces defects?
Not that we've seen. Meta, Cursor and Cloudflare describe their own practice and publish no before/after defect count. METR and SWE-ABS are measured work, but they measure benchmarks rather than shipped products. Several teams independently arrived at the same design. Nobody has proven it works.
What does QA.tech do that my CI doesn't?
It opens your running product on the pull request and exercises it the way a user would, then returns pass, fail or "unable to verify" with screenshots, network and console logs, and a per-step trace. Your CI runs the assertions in your repo. The limits: web is Chromium only, native iOS and Android are separate, and nothing exports as Playwright or code.