Quality Assurance·

Can You Run a Software Factory Without a Verification Layer?

You have CI/CD, review bots and a preview per PR. None proves the product still works after a change. The business case for a verification layer.

QA.tech team

A Verification Layer – a precision inspection gate surrounding a gunmetal component on a tiny industrial production line.

You can, for as long as a person still opens every change and tries it. Past that point, CI, review bots and preview environments show that the code builds, reads well and deploys, and none of them shows the product still works for a user. A verification layer is the independent check that does, by running the product and returning evidence and a verdict.

What a verification layer does that CI/CD and preview environments do not

Nothing in your pipeline is broken. That's what makes this an awkward budget conversation. You already pay for CI/CD, a bot comments on every pull request, each branch gets its own preview URL, and now somebody wants another line item.

This is the business case, for and against. You don't have to call your setup an AI software factory for it to apply. If agents write a growing share of your code and your people mostly review, you're running one.

Everything you already run answers a real question. Look at which one.

What you already runWhat it provesWhat it leaves open
CI: build, lint, types, unit testsThe code compiles and the repository's tests passWhether those tests cover this change. If the agent that wrote the change also wrote the tests, a green run is the author agreeing with itself
Code review botThe diff reads sensiblyAnything that only shows up at runtime. It never opens the deployed app
Human reviewA person understood the diff and accepted the approachMore each month, as pull requests get bigger and arrive faster
Preview environmentThe branch builds and deploys to its own URLEverything, until someone or something opens that URL and uses it
Scripted end-to-end suiteThe flows somebody scripted still workThe flows nobody scripted, including the one this change just added

Read the right-hand column top to bottom and the same gap keeps coming back. Nothing on the list opens the deployed app and does the thing the change was supposed to let a user do.

Where that step still exists, it's a person, usually the reviewer who clicks around the preview before approving, or a tester who gets the link in Slack. Often it's nobody. The preview builds, the URL sits in the pull request, and the approval happens on the diff. If that sounds familiar, the practical fix starts with how to get more out of your preview environments.

Agent-written code makes the first row weaker than it looks. Agents are good at writing unit tests for the task in front of them, and each will probably ship its own slice. Who checks that the slices still work together as one product?

Sometimes it's worse than a gap. A developer's account on Orchid Files of more than 100 hours with one coding agent, published on 4 November 2026, describes what the agent did when a change failed schema validation: "it simply disables schema validation or allows any unknown parameters in the schema. And the tests pass again." That's the short version of why passing tests are not proof.

What a verification layer is

It's a check that sits between "the agent says it's done" and "someone accepts the change". Three properties separate it from one more CI job.

It's independent of the author. The agent that wrote the change doesn't write or edit the checks that judge it. Think of it as a counterparty to the agents producing your code, one that gains nothing when a change passes. Published factory designs keep the author from approving its own work. Gumloop's rule is "The Factory never merges its own work." The Claude Code team at Anthropic draws a similar line. "An agent's approval never counts as a person's," they write, even where automated review is allowed to clear a low-risk change.

It runs the product. The check happens against a deployed instance, a preview or staging, where it signs in and uses the flows the change can reach. Addy Osmani's essay on agentic verification picks a side: "If you had to choose only one test to use, it would be E2E tests."

It returns evidence and a verdict. The verdict is something a pipeline can read: pass, fail, or unable to verify. The evidence is something a reviewer can read in a couple of minutes. A reviewer may not need every line of an agent's diff. What they need is enough to look at the change and say "yes, this makes sense". The same output goes back to the coding agent, so a failure can be fixed before a human opens the pull request.

In the four layers we use for verifying the work of coding agents, this is the fourth: the behaviour of the running product.

What it costs to run without one

You pay for the gap in review queue time and in changes that merge with nobody looking, and later in incidents. What follows comes from teams that publish their numbers.

The factories that work spend heavily on verification

Gumloop sells an agent platform and runs its own engineering on it, so treat its figures as vendor self-reported. As of 28 October 2026, Gumloop's engineering write-up says the company merged 571 agent-written PRs in six weeks, 83% of them with no human code changes. The number comes out of a narrow setup. The work is picked because it "has an obvious, verifiable solution". Every pull request has to arrive "with proof, like a test that fails before the fix and passes after it". An engineer still reviews and merges each one, and the remainder needed that engineer to change the code first.

At Stripe, as of February 2026, over 1,300 pull requests a week merge with no human-written code, and every one is human-reviewed. The last line of verification is a person reading each pull request. That works at Stripe's headcount. It also tells you what's being spent, which is reviewer hours.

Anthropic's Claude Code team reports the outcome most CTOs want: "From June to September, merged pull requests to Claude Code's codebase more than doubled while the share that had to be reverted fell by more than 40 percent." They got there by treating review as a system. An agent reviews every change first, people read the evidence, and merge gates decide how much human review each kind of change gets, a design we go through in who gates the merge. They also name their guardrail: revert and incident rates, watched so that saved review time doesn't come back as breakage.

What the average looks like

In its Q3 2026 telemetry, Faros reports pull requests merged without review, pull request size and monthly incidents all rising, and its own reading is that "review capacity is not growing at the same rate". These are a vendor's numbers from its own telemetry, and they describe volume rising faster than the capacity to check it; the full AI cost of shipping a feature has the figures and what they do to a budget.

What a home-built harness missed

Maple moved its web app off React, 287,000 lines across 131 routes, with an agent doing the port and a home-built harness checking it. The harness drives the old and new apps with Playwright and compares them. It did its job, for the scenarios it had.

After the merge, by Maple's account, a first review found "about 40 defects that happen between screenshots" and a final review by six agents found "about 140 more, four of them serious, in checkout and settings". A harness verifies what someone thought to script. The rest turned up in two later reviews of the merged code, and that gap belongs in any decision on building your QA agents or buying them.

The point where review capacity stops scaling

Line those cases up and the constraint is the same in each. Writing got cheap. Checking didn't.

Human review scales with headcount. Pull request volume now scales with token spend. You can usually tell when the two have crossed. Pull requests wait longer for review than they took to write. Approvals land without anyone opening the preview. Somebody proposes auto-merging "the low-risk ones" and nobody can say which those are. Engineers and their managers start saying out loud that a lot of what ships hasn't been tried by anyone.

From there the choices are few. You can slow the agents down, and some teams do. We've spoken to one that throttled its own agents because each release carried more change than anyone could check. You can let review thin out, which is what the Faros data describes. Or you can add checking capacity that scales the way the writing does.

That last one is what a verification layer buys. The return is confidence to merge at the speed the agents write. A reviewer who gets a verdict with the evidence behind it can approve in minutes and say what was checked. A coding agent that gets a failure with a screenshot can fix it before anyone opens the pull request. It works like insurance on velocity you're already paying for, and on most days it finds nothing serious. That's the expected result.

There's a limit to what this buys. When building gets cheap, building the wrong thing gets more expensive, and a verification layer only tells you the change works. Whether it was worth building is still a product call.

Measure the layer with numbers you'd believe from somebody else: time from pull request to merge, bugs caught before merge, reverts, incidents. Leave "cost avoided" out of the deck. It's always an estimate, and everyone in the room knows it. There's a scorecard to borrow in DORA metrics in the AI era.

Where to place it in the pipeline

Placement decides whether you get a gate or a report, and product quality in the AI SDLC maps where the layer attaches at each stage.

Start on the pull request, against the preview. The change is isolated, its author is still in context whether that's a person or an agent, and the verdict can block the merge. If the coding agent can call the same check before it opens the pull request, it fixes what fails with no human involved.

No preview per pull request? Run the check once the merge has deployed to staging. You'll catch nearly the same things, only later: the change is already on main, so nothing can be blocked, and when staging has taken several merges it's harder to say which one broke it.

Whichever slot you pick, keep a short critical-path suite beside it that runs on every change: sign-up, login, payment, whatever a customer would notice within the hour. Selecting checks from a change is a judgement, and a judgement can miss. The fixed suite is the floor. A scripted regression suite that already runs reliably stays as well. In regulated products you'll be running a full regression pass regardless.

What we built for this slot

Our own version of this layer won't tell you the code is clean. A review reports test results and leaves code-quality opinions to your review tools. It doesn't replace CI or a scripted suite you trust, because scripts are faster and fully deterministic for flows that must run the same way every time. It isn't deterministic either: the selection for one change can differ a little from one review to the next, and the rules your team writes narrow that without removing it. It's a hosted service, and on the web it tests in a Chromium-based browser only.

QA.tech, an AI testing platform, puts a change review on your pull requests. From the PR description, the diff and the linked ticket, the agent decides which of your existing test cases bear on what a user would see change, adds new ones for behaviour the change introduces, and runs the lot against that pull request's preview deployment. Riskier changes get deeper testing, and the size of the diff alone doesn't decide which those are.

Every review ends in one of three verdicts: pass, fail or unable to verify. The third is there so that when the agent couldn't confirm a change, it says so. The pull request carries the verdict and what was tested. The proof is kept in the run the pull request links to: screenshots, logs, the step trace, and a recording if there is one.

On GitHub you can require the review as a check in branch protection, so a failed review blocks the merge. GitLab and Azure DevOps have no such gate for the review, which is posted on the request, so there a test plan that fails your pipeline job can do the gating. For critical paths, pair the review with a small smoke test plan that runs on every pull request. Results are also available over MCP, so a coding agent can start a run, read what failed, fix it and rerun only the failed test cases.

Any shortlist of the best autonomous QA agents for enterprise-scale software engineering teams is a shortlist for this slot. Our entry is dynamic PR testing, summarised for a budget holder on AI testing for CTOs. Hold us to the table below, like anyone else.

What to ask of any verification layer

Bought or built, the layer should pass the same checks. Each row has something you can try in a trial week.

RequirementWhy it mattersHow to check
Independent of the agent that wrote the changeAn author's own tests agree with the authorAsk where the checks live and whether the coding agent can edit them
Uses the running productRuntime failures aren't visible in a diffBreak a user flow on a branch without touching its tests. Is it caught on the preview?
Scopes itself to the change, by riskRunning everything every time is slow and noisy, and a fixed list misses new behaviourOpen a small change to checkout and a large one to a settings page. Compare what was checked and the reason given
Says when it couldn't checkA pass on a flow it never reached is worse than no resultAsk what the verdict is when the check can't reach the changed flow, then ask to see one on a real pull request
Evidence a reviewer can audit in minutesPeople approve on evidence they can readTime a reviewer from the verdict to the screenshot or log of the failing step
A verdict machines can act onAgents fix earlier, and merges can be gatedLook for a required status check on your Git host and an API or MCP endpoint your coding agent can read
A fixed floor for critical pathsSelection is a judgement and can missAsk how sign-up, login and payment get checked on every change, including the ones that never touch them
Low noiseA gate that cries wolf gets bypassedRun it report-only on two weeks of real pull requests and count the failures that weren't real
Cost you can forecastPer-change checks multiply by pull request volumeAsk for a spending cap, then model a month at your real volume

When you can run without one

Yes, you can run without one if the team is small, a person opens every change before it merges, and the agents work under that person's eye. A coding agent testing locally plus a careful reviewer is a verification layer. It's made of people. Small teams have told us as much. For them it's enough for now, and they'll look again once the factory is real. That's a fair call.

It stops being enough when volume passes what those people can open. You'll see it in the signs from earlier, the queue and the approvals on the diff alone. After that the questions are what the whole thing costs and whether to build it or buy it.

FAQ

What is a verification layer?

A verification layer is an independent check that runs your product after a change and returns a verdict with evidence. Independent means the agent or person who wrote the change doesn't write or edit the checks. It sits alongside CI and code review, which examine the code, and it answers a different question: does the product still work for a user?

Can you run a software factory without a verification layer?

For a while, yes. While a person still opens and tries every change, people are the verification layer. Once agents open more pull requests than reviewers can try by hand, review queues grow or review gets skipped. The published factories that hold up, such as Gumloop's and Stripe's, keep a human review on every pull request.

If we already have CI/CD and preview environments, why add a verification layer?

CI proves the code builds and the repository's tests pass. A preview proves the branch deploys. Neither one uses the product. Someone or something still has to open the preview and try the flows the change touches, and at agent volume that someone runs out of hours.

How do AI agents verify their own code changes?

Mostly by writing and running tests for the task they're working on, then reading the result. That catches a lot. It also means the author is grading its own work, and agents have been documented weakening a check to get a green run. Teams that depend on agent-written code add a second, independent check.

Is an AI software factory safe without verification?

At low volume it can be, because people still look at everything. At high volume the published data points the other way. In its Q3 2026 dataset, Faros reports monthly incidents rising as the number of changes climbs. Anthropic's Claude Code team, which reports merging more while reverting a smaller share, watches revert and incident rates as its guardrail.

What is code verification in the context of software quality?

Code verification checks that a change meets its specification, using static analysis, type checks, unit tests and review. It's necessary, and it works on the code. Whether the running product does what a user needs is a separate check. Standards call that validation, and it's the job of a verification layer, so the difference between verification and validation is worth getting straight before you name the budget line.

Your team moves fast. Can your testing keep up?

Our QA agents test your product autonomously, so moving fast never means shipping broken. Start free or get a 30-minute demo.