Coding agents can open more PRs in a day than your team can review in a week. An AI software factory is supposed to turn that agentic output into shipped features. In practice, it moves the bottleneck from writing code to checking it.
TL;DR: An AI software factory is many coding agents working in parallel loops, fed by a queue of specs. Generation scales with compute, but verification does not, so the amount of autonomy you can safely give your agents depends on how well you can verify the running app. Unit tests and AI review bots check the code; something independent still has to check the product a user sees.
Below are the eight blocks, the tools for each, and what breaks when you leave one out.
What Is an AI Software Factory?
An AI software factory is a system of coding agents running in parallel loops, fed by a queue of specs and tickets, where every change passes through automated checks and a review gate before it reaches production.
The definition most people use comes from Addy Osmani, and it rests on three ideas:
- A loop is one agent doing one job on repeat: gather context, make a change, check the result, and go again.
- A harness is everything around that loop: the sandbox it runs in, the tools it can reach, its memory, and the rules for what "done" means. What a test harness is covers the version coding agents need.
- A factory is many harnessed loops running at once, with humans owning the whole thing from above.
How much humans still read depends on how far along you are. Dan Shapiro's Five Levels, published in January 2026, give a shared scale. The names and the right-hand column are our summary of his descriptions:
| Level | Name | Who reads the code |
|---|---|---|
| 0 to 2 | Spicy autocomplete to pair programming | A human writes or approves the code |
| 3 | Developer as manager | A human reviews the agent's diffs |
| 4 | Developer as product manager | A human writes the spec and checks that the tests pass |
| 5 | Dark factory | Nobody reads the code |
Osmani adds a useful split. In a light factory, humans stay in the loop at design and review time. In a dark factory, code ships that no human has read, and only machines verify it. We covered who else uses the term, and what they call it, in what an AI software factory is and where verification sits in it.
What Are the Building Blocks of an AI Software Factory?
Here is the full set, with what each block does and what happens when you leave it out:
| Building block | What it does | Example tools | What breaks without it |
|---|---|---|---|
| Work queue and specs | Turns intent into tasks an agent can build | Linear, Jira, GitHub Spec Kit | Agents build the wrong thing, quickly |
| Coding agents and harness | Writes the change inside limits | Claude Code, Codex, Cursor | Loops wander, stall, or edit what they should not |
| Shared context | Rules files and Model Context Protocol (MCP) servers | AGENTS.md, CLAUDE.md, MCP | The same mistakes repeat on every PR |
| Preview deployments | Gives every PR its own running build | Vercel, Netlify | There is nothing real to verify before merge |
| Code-level checks | Unit tests, continuous integration (CI), static analysis, AI review bots | GitHub Actions, CodeRabbit | Broken code merges |
| Running-app verification | Checks user journeys in the running app, in a browser or a mobile build | QA.tech, scripted end-to-end suites | Correct code ships a broken product |
| Risk-tiered review gate | Sends each change to the cheapest review that is still safe | Branch protection, merge queues | Reviewers drown, or rubber-stamp |
| Rollback and production signals | Undoes bad changes and feeds incidents back into the queue | Feature flags, observability tools | Escaped bugs pile up unnoticed |
A normal pipeline with CI and previews already has the first five. The sixth is the one the factory blueprints name and rarely specify, and it is where the verification layer of the factory lives.
What Does Running-App Verification Look Like in Practice?
Say a coding agent ships a redesigned checkout flow. Unit tests pass, the review bot leaves two style comments, and CI is green. On the preview deployment, though, the new layout pushes the saved-card selector below a sticky banner, so returning customers cannot select a card. No line of the diff looks wrong, and after merge every returning customer hits it.
This is the gap QA.tech, an AI testing platform, is built for. It won't catch everything; what it removes is the class of bug that only shows up when someone opens the app. Its QA agents receive a user goal ("pay with a saved card and see the confirmation") and reason about the live app the way a user would. The test case is written as a goal, so a renamed selector doesn't break it. A web or mobile test case can also check email and SMS and make API calls along the way, so the checkout test case can confirm that the receipt email arrives.
In a factory, verification hooks in at two points:
- Before a PR exists. The coding agent calls the MCP server from inside Claude Code, Cursor, or Codex. It starts a run, reads the result, and fixes the failure before anyone opens a PR. The MCP docs list the tools available, such as
start_run_with_test_casesandget_run_test_case_trace. - On the PR. PR testing runs against the PR's preview deployment, or against staging straight after merge if you have no previews. It selects the existing test cases the change affects and creates new ones only where coverage is missing. It then posts the result on the PR: pass, fail, or "unable to verify", with a status check you can require before merge. How to verify the work of coding agents walks through what the agent reads and selects on each PR.
When Does a Dark Software Factory Not Make Sense?
For most teams, a fully dark factory is not the right target. Osmani warns about comprehension debt, which he defines as "the widening gap between how much code exists and how much any human still understands." A dark factory adds to that debt as fast as it can, with the tests green the whole way. That risk is highest in large, decade-old systems, which is where a lot of mid-market and enterprise teams work.
A light factory is often the better choice when:
- The domain is regulated, and a human must sign off on intent.
- The codebase is old and poorly understood, so humans need to keep reading in order to keep learning.
- Usability and product judgment matter more than raw throughput.
In every case, QA agents take over the repetitive verification while people keep the judgment calls. Our co-founder and Chief AI Officer Vilhelm von Ehrenheim made the point in his GOTO Copenhagen talk that ten times the code doesn't mean ten times the testing: any change can regress any dependency, so verification demand grows faster than the code does. For test strategy, we recommend a small, pinned regression set for your critical paths, with dynamic testing on every PR and exploratory testing covering everything else. Rebuilding your old suite one to one only moves the old brittleness into a new tool.
Where Should You Start in 2027?
Build the blocks in order of what each one unlocks:
- Give every PR a preview deployment, so there is something real to verify before merge. Vercel previews are the common starting point. If you can't yet, verify on staging straight after merge.
- Verify the running app on every PR, using a verifier that did not write the change.
- Add risk tiers to your review gate, so low-risk changes with clean evidence can merge on the check alone and high-risk changes go to a human who starts from the evidence.
- Raise agent autonomy only as fast as your verification can keep up.
If you would rather not build and maintain step two yourself, the PR testing and MCP setup described above covers it.
FAQ
What is an AI software factory? An AI software factory is a system of coding agents running in parallel loops, fed by a queue of work, with every change passing automated checks and a review gate before production.
What is the difference between a light and a dark software factory? A light factory keeps humans in design and review, while a dark factory ships code that no human has read and that only machines have verified.
What is the biggest bottleneck in an AI software factory? Verification is the biggest bottleneck, because generating code scales with compute while checking that the product works has traditionally scaled with human attention.
Do AI code review bots replace end-to-end testing? No, because review bots read the diff but never open the app, so they cannot confirm that a user journey works.
Can a coding agent verify its own pull request? A coding agent can run a first pass on its own work, but the final check should come from a verifier that wrote neither the code nor its tests.
Do I still need unit tests in an AI software factory? Yes. Unit tests are still the fastest layer. They check the code, and running-app verification checks the product.
Will QA agents replace QA engineers? No. QA agents take the repetitive checking. QA engineers move to writing acceptance criteria and making the risk calls on what the agents should verify.
