What Is Autonomous Testing? Levels, Reality Check, Where Agents Fit

Autonomous testing is software testing where the system works out how to test, runs it, judges the result and reports back, and a person's job shrinks to setting goals and reviewing verdicts. The further version, where the system also decides what to test, is where the industry is heading and is not shipping yet. In 2026, agents can run end-to-end tests from a plain-language goal without a script, which removes most of the human work that used to sit in authoring and maintenance. Nobody has removed the human who decides what "correct" means, and you should be suspicious of any vendor who says they have.

The short version

We build an autonomous testing platform at QA.tech, so read this knowing who wrote it. The levels are drawn to be checkable against any vendor, including us, and the section on what no agent does is the one we'd most like you to read.

Autonomous, agentic and automated are not synonyms

Automated testing means a machine executes steps a person wrote. Playwright, Cypress, Selenium. The machine does the clicking; every decision was made in advance by whoever wrote the script.

Agentic testing means an AI agent reasons its way to a goal at runtime instead of replaying steps. It describes how the test runs. What is agentic testing covers the mechanics.

Autonomous testing describes how many decisions the system makes. A scripted suite that runs nightly with nobody watching is unattended, which is a different thing: nobody is in the room, but every decision was made by a person months ago, and the suite is useless the day the UI changes. Autonomy counts decisions the system makes at runtime, not people absent from the room. So it's a scale, and the useful question is where a tool sits on it and where you need it to.

The term also predates agents. Around 2018 to 2020 "autonomous testing" was attached to machine-learning features inside script-based tools: test-impact analysis, element locators that adapted, crawlers that recorded flows. Readers who met the word then arrive expecting something much weaker than what a goal-driven agent does now, which is one reason the levels below are worth spelling out.

The levels, zero to five

This is really two ladders joined in the middle, and it's clearer to say so. Levels 0 to 3 describe how the test gets produced. Levels 4 and 5 describe who makes the decisions at runtime, which is the axis autonomy is actually about. At each level, count the moments a person has to stop and think: write the assertion, fix the selector, decide whether the failure is real, decide what to test at all. Levels 1 to 3 mostly move where that thinking happens. Levels 4 and 5 remove some of it. A vendor can climb three rungs on this ladder without reducing your team's workload by an hour.

Level 0: manual. People click through the product before release. Slow, thorough in the hands of a good tester, impossible to scale with release frequency. Still where a surprising number of teams are for anything outside the happy path.

Level 1: scripted automation. Engineers write test code. It runs on its own, breaks when the UI changes, and needs an engineer to fix it. Most teams with a CI pipeline are here. One director of QA counted it for us: 30 to 40% of his team's week went into keeping existing automation alive. One team, one quarter, so treat it as a shape rather than a benchmark, but nobody we've asked has answered "almost none".

Level 2: assisted scripting. Something else writes the script. Claude and Cursor generate it from a description; Playwright codegen records it from your clicks. The output is a script either way, so the maintenance bill doesn't change. The bottleneck moves from authoring to upkeep.

Level 3: self-healing scripts. The script repairs its own selectors when they break. Maintenance drops, and in exchange your safety net now contains a guess that nobody reviewed. Worth knowing which of your green tests healed themselves last week, and most teams running this can't tell you.

Level 4: goal-based agents. No script. A person writes a goal; an agent plans, acts and verifies against the live app on every run, and adapts when the interface moves. Authoring is a sentence, selector maintenance disappears, and the agent can judge outcomes nobody asserted. A person still writes the goals and reads the verdicts. Several vendors ship at this level today. QA.tech is one: an AI QA testing platform whose agents run plain-language goals against web and native mobile apps, on every pull request and on a schedule, with no scripts or selectors to maintain. If you're evaluating, the useful question is not who claims Level 4 but what the unit of authorship is, because a generated script is still a script.

Concretely, a Level 4 test looks like this. The goal: "Sign up as a new user on the trial plan, add two seats, and confirm the invoice preview shows the prorated amount. Proration is daily, 30-day month, seats are 24.00 each." No selectors, no waits, no page objects. On each run the agent finds its own way to the signup form even if the form moved, and the verdict is a sentence rather than a boolean: "Invoice preview showed 48.00; the proration rule in the goal gives 30.40 for 19 remaining days on a two-seat upgrade." Note where the rule came from. The agent found the form on its own and read the invoice on its own, but the pricing policy was in the goal, because it lives in a spreadsheet in finance and not in the product. That division of labour is the whole subject of the next section.

Level 5: policy-driven. You stop specifying tests at all. You specify risk: this flow is revenue-critical, this one can break for a day, this compliance path must never regress. The system decides what that implies, keeps the coverage honest as the product changes, and escalates only the verdicts a human has to own. The difference from Level 4 is not that the system proposes tests, which several tools including ours already do. It's that nobody is maintaining a list. Parts of this exist, and yes, the parts that exist map closely onto what we and a few others ship: agents that crawl an app and propose tests, agents that read a pull request and pick the flows to run, coverage maps that show what has never been exercised. Nobody ships the whole level. Two pieces of it may never be engineering problems: deciding which risk a flow actually carries, and deciding that a passing test was testing the wrong thing. Those are product judgement, and the next section is about why they stay with you.

What no agent takes off your plate

Deciding what correct means. An agent derives its plan from the goal and the application. If the application is wrong and the goal was derived from the application, the test confirms the wrong behaviour. Goals have to come from what the product should do rather than what it currently does, which means tickets, acceptance criteria, and somebody's understanding of the business. Tools that read the issue tracker help. On a product with real domain rules, tax, eligibility, pricing tiers, supplying those rules is most of the work, and it stays with you.

Noticing what's missing. An engineering lead whose team went a different way after a trial put it in a way that stuck: the regressions that hurt them were "things that weren't being built", features that should have existed and didn't. No agent exploring an app will find a feature that isn't there. Neither will a script.

Reading the verdict. An autonomous system produces verdicts, and someone decides what to do with them. During one pilot the customer came back with a complaint: the agent had "approved test cases we don't really feel should have been approved." They wanted to see the reasoning on every verdict after that, and they were right to. The audit trail decides whether a wrong verdict costs you a minute or a release.

So the target for most teams is Level 4 with a person still in the loop at two points: writing the goals, and owning the verdicts that matter. Budget for the second one. Teams plan for the authoring time they're saving and not for the review time they're taking on.

Three practical constraints

Agent runs are slower and more expensive per test than a Playwright script, because you're paying for reasoning rather than replay. That's why running the whole suite on every commit is the wrong shape: you select the affected flows on the pull request and keep the scheduled full pass for the nightly. Pricing in this category is per unit of work (QA.tech bills per test-case execution; others per step or per token), so ask what the same goal costs on run one and run ten, and whether a re-run after a flake bills again.

An agent also needs somewhere real to run and something real to log in as. Test data, seeded accounts, a preview environment per branch, an SSO or two-factor path a machine can get through, a payment sandbox, cleanup after a run that creates records. None of that is new to end-to-end testing and none of it is solved by autonomy, and in most evaluations we do it's the work that decides whether week one goes well. Ask any vendor how their agent authenticates against your login before you ask anything about their model. Ask where the screenshots go while you're at it: an agent on your staging environment is sending page content to a model provider, so you want the providers, the regions and the retention terms in writing. Ours are SOC 2 Type 2, data at rest in the EU, zero-retention model terms, and a requirement that test data be synthetic, which is a constraint on you rather than a feature of ours.

And autonomous testing is the wrong tool for a real set of jobs. Unit tests, contract tests, load and performance work, pixel-exact visual diffs, and anything where a regulator wants a byte-identical audit of the same steps every quarter. Those want determinism, and determinism is the thing you traded away.

What changes when you get there

The queue goes away. Tickets sitting in "ready for test" waiting for a person are the most common symptom we hear about, on teams with plenty of automation as well as teams with none. Autonomous execution on every pull request removes the wait because the verdict arrives while the branch is still open.

The suite stops growing for its own sake. Scripted suites grow because deleting a test feels dangerous and nobody remembers what each one guards. Goal-based tests are cheap to write and cheap to reason about, so you can keep the scheduled suite small and let dynamic testing on the pull request cover what changed. What regression testing is for makes that case in full. The existing scripted suite doesn't get deleted on day one: run both for a quarter, retire scripts one flow at a time as the agentic version proves itself, and expect to keep some scripts permanently.

End-to-end testing becomes viable at all. A senior engineer at a large travel company told us his teams had historically avoided end-to-end tests "because it was brittle", and that agents maintaining the flows was what made the layer worth building.

Coverage becomes visible, up to a point. Because the agent has mapped the app, "which of these flows have we never exercised" has an answer for the first time. It still can't tell you about the flow that was never built, which is the harder gap and stays yours.

How to evaluate an autonomous testing platform

Six questions, in the order we'd ask them.

1. Where on the levels is it, really? Ask what the unit of authorship is. If it's a script, even a generated one, it's Level 2 or 3 whatever the landing page says.
2. What happens when the UI changes incorrectly? Adapting is the pitch. What stops it adapting around a bug? The answer should involve goals that are independent of the UI and a verdict you can audit.
3. Is it deterministic? It shouldn't claim to be. Agents are statistical. Where there is one obvious path through a flow, runs look stable in practice. Where there are two, you'll see both. Ask what the run-to-run variance looks like on a suite your size, what happens on a retry, and how you're told when the model underneath the agent changes, because providers retire models and no vendor controls that calendar. A good answer sounds like: re-run a red once automatically, block only on two consecutive fails, quarantine anything above your existing flake tolerance.
4. What does the verdict come with? This is cheap or expensive depending on what arrives with the red. A screenshot, the agent's reasoning and the failing request means thirty seconds. A red X means you re-run it by hand.
5. What does it remember? Ask how the platform knows your app on the hundredth run. If every run starts from zero, cost and speed scale badly.
6. How does it fit the pipeline? Pull-request verdicts, CI triggers, an MCP server for the coding agents your developers already use. If the verdicts only exist in the vendor's dashboard, nobody on the team will look at them.

Autonomous testing at QA.tech

QA.tech is built for Level 4 and reaching into 5. Tests are goals in plain language. Autonomous QA agents map your application into a knowledge graph, run flows in a real browser or mobile emulator, judge the outcome every run, and leave screenshots, video, logs and their own reasoning as evidence. On every pull request the platform reads the diff, picks the affected flows, runs them against the preview and posts a verdict to GitHub or GitLab before merge. The agent proposes tests from a crawl of your app and from your tickets, and coverage tracking (in beta) shows what has never been exercised.

Two limits. The pull-request integration uses the diff only to work out which flows are affected, never as the source of truth for what the app should do, so a bug that ships looking deliberate is a bug it can still miss. And it isn't deterministic, which rules it out of the byte-identical audit case above. You choose the agent per test (fast for stable flows, more capable for mobile and complex ones); the models underneath are ours to keep current, and we'd rather you knew that too. Our position on self-healing, which we don't do, is in the top 5 things everyone gets wrong about self-healing tests.

For how the agents work step by step and the questions to ask any vendor, see the AI QA testing guide.