Ai·

What is an AI software factory, and where does verification sit in it?

An AI software factory is a delivery system where agents build, test and ship code and humans design it. Gartner, BCG, Stripe and OpenAI describe one; none specifies the QC station.

QA.tech

An AI software factory is a software delivery system in which AI agents do the building and humans design the system the agents work inside. Intent goes in at one end as a ticket, a spec or a bug report. Agents plan the change, write it, test it, review it and ship it. People set the rules, the checks and the acceptance criteria, and they review what came out. The phrase is decades old (Michael Cusumano's 1991 book Japan's Software Factories studied a much tamer version); what changed in 2026 is that Gartner, BCG, Stripe, OpenAI and the tooling vendors selling into them started using it for the same shape at the same time.

This page covers what they mean, what else people call it, and the one station every blueprint names and almost none of them specifies: quality control.

Who is saying "software factory" in 2026

Gartner published Innovation Insight: AI Software Factories on 17 September 2026, calling the factory the "North Star" delivery system for agentic software engineering, and arguing that without one, agents sit trapped inside a slow, human-coordinated SDLC. BCG Platinion's The Agentic Software Factory, from March, defines it as autonomous agents that build, test and ship software around the clock, and reports 3 to 5x productivity gains from the organisations running one. Factory 2.0, the June relaunch from the company that took the name, describes an end-to-end system where signals become changes that are built, tested, reviewed, secured and shipped in a loop that observes itself.

Two engineering teams published what theirs actually look like. OpenAI's harness engineering post, from February, describes a product shipped with zero hand-written code, around a million lines in five months, where humans may review pull requests but aren't required to. Stripe's Minions run the other model: more than a thousand pull requests a week, written start to finish by the agents and reviewed by humans. Our co-founder Vilhelm von Ehrenheim spoke on this at GOTO Copenhagen on 2 October, in a talk titled Nobody Reads the Code Anymore: Quality Engineering for the Software Factory.

The search data lags the conversation. "Software factory" draws about 720 searches a month in the US and "what is a software factory" another 110; "ai software factory" is small but has gone from near zero to its peak in three months (Semrush, US, October 2026). People are hearing the phrase in analyst decks and conference talks and going to look it up.

What people call it

The same idea runs under several names, and the names say something about who's using them.

Software factory is the analyst and vendor term: Gartner, Faros, Overcut, Augment. Factory.ai uses it as a product name.

AI software factory and agentic software factory are the consultancy forms: Gartner for the first, BCG and TrueFoundry for the second. Same thing with the agent made explicit.

Dark factory is the version with no humans reading code. StrongDM's CTO set it out in February as two rules, code must not be written by humans and code must not be reviewed by humans, and the rest of the system follows from that. Simon Willison's write-up, borrowing Dan Shapiro's "dark factory" label, made the name stick: a team that builds serious software without even looking at the code. A consultancy has since branded its own delivery model a multi-agent dark factory.

Lights-on factory is re:cinq's term for the other pole, and Stripe is its example: agents write everything, people still review everything. The re:cinq blueprint is explicit that the factory is a system where agents produce the code and humans design the system those agents operate within.

AI factory mostly means something else. In search it's NVIDIA's term for computing infrastructure that runs the whole AI lifecycle, with intelligence as the product and tokens as the unit, so it's not a synonym. But in-house teams do use it for their own agentic setup: a team running about sixty parallel agent instances through 2,000-plus issues in a month called its write-up "Building an AI Factory".

Agent factory is how Microsoft's Jay Parikh talks about it. The EVP of Core AI described the goal as putting the pieces together into a production line.

If you hear any of these in a meeting, they're describing the same loop with the human dial set differently.

The shape of the loop

Every blueprint, whatever it's called, has the same five stations.

  1. Intent. A ticket, a spec, a bug report, a customer signal. In the factory framing this is the raw material, and the quality of the output tracks the quality of the intent.
  2. Build. One or more agents plan and write the change. Stripe's run one-shot, end to end; OpenAI's run many in parallel with their own worktrees and app instances.
  3. Checks. Everything that runs on the change before a human sees it: lint, types, structural tests, the agent's own tests, review agents, security scans.
  4. Ship. Merge and deploy, with the human dial somewhere between "reviews every PR" and "reviews none".
  5. Observe. Production signals feed back as new intent. Factory.ai's phrasing is that the factory improves by observing itself.

The people move to different stations. In the BCG version they define intent and review outcomes. In re:cinq's they design the system. In Vilhelm's talk, QA moves up the loop into acceptance criteria, product risk analysis, writing invariants and calibrating the judges, and delegates the execution down to agents.

The station every blueprint names and none of them specifies

Read the blueprints for how station 3 works and the detail runs out quickly. Factory.ai says changes are tested, reviewed and secured. TrueFoundry names "automated quality control" once, in an FAQ answer, and never defines it; the checks it does name are repository tests, static analysis, CI checks, sandbox and egress controls, merge policy and connected evaluators. Not one of them opens the product. Faros lists automated tests, static analysis, security scanning, review agents, human review and production signals. BCG gets closest: it says trust must be engineered into the system, proposes scenario-based tests run by independent agents, and specifies that the behavioural scenarios are stored outside the codebase the agents can reach. StrongDM does the same with its Scenarios, end-to-end user stories it stores outside the codebase, in its own words like a holdout set in model training, and validates with a model.

The common list is tests the agent wrote, checks on the code's shape and a reviewer that reads the diff. Nothing in it opens the product and uses it. The two teams that went furthest toward lights-out, StrongDM and BCG's model, both reached for the same thing: behavioural scenarios the agent can't edit, checked by something that isn't the agent. They built the QC station even though the definitions they were working from leave it out.

Faros's Acceleration Whiplash report, from telemetry on 22,000 developers across 4,000 teams, found higher AI adoption associated with 33.7% more task throughput and 66.2% more epics completed, alongside 54% more bugs per developer, incidents per PR up 242.7%, and median PR review time up about fivefold. Volume went up, the checks didn't keep pace, and the gap landed on the reviewers. At GOTO Copenhagen, half of the 440 engineers polled during the DORA keynote named reviewing changes as the biggest friction point in their SDLC.

Daniel Mauno Pettersson, our CEO, said in a webinar in June that everybody expects to move faster with agentic coding, and that this means stepping up the process around it: testing, monitoring, observability, review, everything that goes into the SDLC. Vilhelm's version from the talk is that ten times the code doesn't mean ten times the testing. Any change can regress any dependency, so verification demand scales worse than linearly. The factory metaphor, taken seriously, agrees: no factory ships without inspection, and the ones that work build it into the line.

Where verification sits in the factory

Here is where we think the QC station goes, and what it has to be.

At the pull request, against a running instance. Verification of behaviour has to happen where the new code is reachable as a product: the preview before merge, or staging straight after it when there is no preview. The further it sits from the change, the closer it gets to end-of-line inspection with the defect already in the line.

Run by something that didn't build the change. BCG stores its scenarios outside the codebase the agents can reach, and StrongDM keeps its Scenarios outside the codebase too, because the agent's own tests are a weak judge of its own work. A verifier that's separate from the doer is the pattern OpenAI and Meta settled on for agent actions; it applies to products too.

Scoped by the change. A factory doesn't re-inspect the whole warehouse for every pallet. The verifier reads what the change claims to do, works out which user-facing flows it can affect, and checks those. The regression suite stays small; the per-change verification is where the coverage lives.

Reporting three states. Pass, fail with evidence, and a third for runs that never reached the change. A factory QC station that reports "no defects found" when the inspector never looked at the part is worse than none.

Feeding back to the agent. The verdict should reach the coding agent as well as the reviewer, so a failing trace gets fixed before anyone opens the PR.

This is the job dynamic PR testing does at QA.tech, an AI testing platform. It won't catch everything: review is the floor, it catches bad code, and the verifier catches broken products. It runs goal-based scenarios against the preview, or against staging after merge where there is no preview, reports the three states above, with screenshots of the steps the assessment cites and the network and console logs behind every run, and the coding agent can call it over MCP for a release check on its own changes. The merge stays a person's call even when the verdict is wired in as a required status check. How to verify the work of coding agents walks through what the agent reads and selects on each PR.

Getting there from a normal pipeline

No team we've spoken to this year is starting from a dark factory. The starting point we see is a normal pipeline where agents write a growing share of the code, review has become the queue, and QA headcount hasn't moved. A marketplace CTO in the Nordics put it as token spend up fivefold since spring and review as the thing that now blocks.

The order we'd suggest:

  1. Encode the things you keep correcting by hand as static checks the agent runs itself: lint, types, structural rules.
  2. Write your behavioural scenarios as goals, in plain language, outside the repo the agents edit. BCG and StrongDM both do this, and it is what lets a scenario outlive the UI it was written against.
  3. Put a verifier at the PR that runs those scenarios against the preview and reports three states. Let the coding agent see its output.
  4. Measure the judges. Calibrate the review agent and the verifier against a hand-labelled sample and watch precision over time, so a change in the model or the prompt shows up as a number.
  5. Move the human dial as the numbers allow. Auto-merge the low-risk class first and keep a person on anything user-facing.

FAQ

What is an AI software factory?

A software delivery system in which AI agents plan, write, test, review and ship changes, and humans design the system: the intent, the rules, the checks and the acceptance criteria. Gartner (September 2026), BCG Platinion (March 2026) and Factory.ai (June 2026) each describe it this way with different emphasis. The 1990s "software factory" meant standardised tooling and reuse; the 2026 meaning assumes agents do the building.

What is a dark factory in software?

A software factory where no human writes or reviews code. StrongDM set it out as two rules in February 2026; Dan Shapiro named the level and Simon Willison's write-up spread it. The opposite pole, where agents write everything and humans still review everything, is what Stripe runs.

Is a software factory the same as the agentic SDLC?

Nearly. "Agentic SDLC" names the lifecycle with agents in each phase; "software factory" names the delivery system built around it and emphasises throughput and the human designing the system. We use factory for the system and agentic SDLC for the phases.

Does an AI software factory need human review?

It needs human judgement on outcomes. Whether that includes reading every pull request is the dial each team sets. Stripe's answer is yes, OpenAI's internal team's answer is not required, StrongDM's is no. All three run automated checks before any human is involved.

Where does QA sit in a software factory?

Inside the loop, at the pull request, as a verifier that exercises the product in a preview before merge, or in staging after it when there is no preview. QA's human work moves up to writing acceptance criteria and behavioural scenarios and calibrating the automated judges.

What is the difference between a software factory's tests and its verification?

In the blueprints above, the tests are the agent's own unit tests plus static checks, which confirm the code is well-formed and does what the agent thought it should. Verification checks the behaviour of the running product against the intent, by a separate agent or person. How to verify the work of coding agents covers the four layers.

Sources

View as Markdown

Your team moves fast. Can your testing keep up?

QA.tech agents test your product autonomously, so moving fast never means shipping broken. See how it works in a 30-minute demo.

Book a technical session