Ai·

AI Software Factory Building Blocks for 2027

The eight building blocks of an AI software factory, the tools for each, and what breaks when you leave one out. Verification sets the limit on autonomy.

QA.tech

Interlocking acrylic and gunmetal building blocks joined with brass around a glass inspection chamber, illustrating an AI software factory.

Coding agents can open more PRs in a day than your team can review in a week. An AI software factory is supposed to turn that agentic output into shipped features. In practice, it moves the bottleneck from writing code to checking it.

TL;DR: An AI software factory is many coding agents working in parallel loops, fed by a queue of specs. Generation scales with compute, but verification does not, so the amount of autonomy you can safely give your agents depends on how well you can verify the running app. Unit tests and AI review bots check the code; something independent still has to check the product a user sees.

Below are the eight blocks, the tools for each, and what breaks when you leave one out.

What Is an AI Software Factory?

An AI software factory is a system of coding agents running in parallel loops, fed by a queue of specs and tickets, where every change passes through automated checks and a review gate before it reaches production.

The definition most people use comes from Addy Osmani, and it rests on three ideas:

  • A loop is one agent doing one job on repeat: gather context, make a change, check the result, and go again.
  • A harness is everything around that loop: the sandbox it runs in, the tools it can reach, its memory, and the rules for what "done" means. What a test harness is covers the version coding agents need.
  • A factory is many harnessed loops running at once, with humans owning the whole thing from above.

How much humans still read depends on how far along you are. Dan Shapiro's Five Levels, published in January 2026, give a shared scale. The names and the right-hand column are our summary of his descriptions:

LevelNameWho reads the code
0 to 2Spicy autocomplete to pair programmingA human writes or approves the code
3Developer as managerA human reviews the agent's diffs
4Developer as product managerA human writes the spec and checks that the tests pass
5Dark factoryNobody reads the code

Osmani adds a useful split. In a light factory, humans stay in the loop at design and review time. In a dark factory, code ships that no human has read, and only machines verify it. We covered who else uses the term, and what they call it, in what an AI software factory is and where verification sits in it.

What Are the Building Blocks of an AI Software Factory?

Here is the full set, with what each block does and what happens when you leave it out:

Building blockWhat it doesExample toolsWhat breaks without it
Work queue and specsTurns intent into tasks an agent can buildLinear, Jira, GitHub Spec KitAgents build the wrong thing, quickly
Coding agents and harnessWrites the change inside limitsClaude Code, Codex, CursorLoops wander, stall, or edit what they should not
Shared contextRules files and Model Context Protocol (MCP) serversAGENTS.md, CLAUDE.md, MCPThe same mistakes repeat on every PR
Preview deploymentsGives every PR its own running buildVercel, NetlifyThere is nothing real to verify before merge
Code-level checksUnit tests, continuous integration (CI), static analysis, AI review botsGitHub Actions, CodeRabbitBroken code merges
Running-app verificationChecks user journeys in the running app, in a browser or a mobile buildQA.tech, scripted end-to-end suitesCorrect code ships a broken product
Risk-tiered review gateSends each change to the cheapest review that is still safeBranch protection, merge queuesReviewers drown, or rubber-stamp
Rollback and production signalsUndoes bad changes and feeds incidents back into the queueFeature flags, observability toolsEscaped bugs pile up unnoticed

A normal pipeline with CI and previews already has the first five. The sixth is the one the factory blueprints name and rarely specify, and it is where the verification layer of the factory lives.

What Does Running-App Verification Look Like in Practice?

Say a coding agent ships a redesigned checkout flow. Unit tests pass, the review bot leaves two style comments, and CI is green. On the preview deployment, though, the new layout pushes the saved-card selector below a sticky banner, so returning customers cannot select a card. No line of the diff looks wrong, and after merge every returning customer hits it.

This is the gap QA.tech, an AI testing platform, is built for. It won't catch everything; what it removes is the class of bug that only shows up when someone opens the app. Its QA agents receive a user goal ("pay with a saved card and see the confirmation") and reason about the live app the way a user would. The test case is written as a goal, so a renamed selector doesn't break it. A web or mobile test case can also check email and SMS and make API calls along the way, so the checkout test case can confirm that the receipt email arrives.

In a factory, verification hooks in at two points:

  1. Before a PR exists. The coding agent calls the MCP server from inside Claude Code, Cursor, or Codex. It starts a run, reads the result, and fixes the failure before anyone opens a PR. The MCP docs list the tools available, such as start_run_with_test_cases and get_run_test_case_trace.
  2. On the PR. PR testing runs against the PR's preview deployment, or against staging straight after merge if you have no previews. It selects the existing test cases the change affects and creates new ones only where coverage is missing. It then posts the result on the PR: pass, fail, or "unable to verify", with a status check you can require before merge. How to verify the work of coding agents walks through what the agent reads and selects on each PR.

When Does a Dark Software Factory Not Make Sense?

For most teams, a fully dark factory is not the right target. Osmani warns about comprehension debt, which he defines as "the widening gap between how much code exists and how much any human still understands." A dark factory adds to that debt as fast as it can, with the tests green the whole way. That risk is highest in large, decade-old systems, which is where a lot of mid-market and enterprise teams work.

A light factory is often the better choice when:

  • The domain is regulated, and a human must sign off on intent.
  • The codebase is old and poorly understood, so humans need to keep reading in order to keep learning.
  • Usability and product judgment matter more than raw throughput.

In every case, QA agents take over the repetitive verification while people keep the judgment calls. Our co-founder and Chief AI Officer Vilhelm von Ehrenheim made the point in his GOTO Copenhagen talk that ten times the code doesn't mean ten times the testing: any change can regress any dependency, so verification demand grows faster than the code does. For test strategy, we recommend a small, pinned regression set for your critical paths, with dynamic testing on every PR and exploratory testing covering everything else. Rebuilding your old suite one to one only moves the old brittleness into a new tool.

Where Should You Start in 2027?

Build the blocks in order of what each one unlocks:

  1. Give every PR a preview deployment, so there is something real to verify before merge. Vercel previews are the common starting point. If you can't yet, verify on staging straight after merge.
  2. Verify the running app on every PR, using a verifier that did not write the change.
  3. Add risk tiers to your review gate, so low-risk changes with clean evidence can merge on the check alone and high-risk changes go to a human who starts from the evidence.
  4. Raise agent autonomy only as fast as your verification can keep up.

If you would rather not build and maintain step two yourself, the PR testing and MCP setup described above covers it.

FAQ

What is an AI software factory? An AI software factory is a system of coding agents running in parallel loops, fed by a queue of work, with every change passing automated checks and a review gate before production.

What is the difference between a light and a dark software factory? A light factory keeps humans in design and review, while a dark factory ships code that no human has read and that only machines have verified.

What is the biggest bottleneck in an AI software factory? Verification is the biggest bottleneck, because generating code scales with compute while checking that the product works has traditionally scaled with human attention.

Do AI code review bots replace end-to-end testing? No, because review bots read the diff but never open the app, so they cannot confirm that a user journey works.

Can a coding agent verify its own pull request? A coding agent can run a first pass on its own work, but the final check should come from a verifier that wrote neither the code nor its tests.

Do I still need unit tests in an AI software factory? Yes. Unit tests are still the fastest layer. They check the code, and running-app verification checks the product.

Will QA agents replace QA engineers? No. QA agents take the repetitive checking. QA engineers move to writing acceptance criteria and making the risk calls on what the agents should verify.

Your team moves fast. Can your testing keep up?

Our QA agents test your product autonomously, so moving fast never means shipping broken. Start free or get a 30-minute demo.