Contents
AI in software testing means using models and agents to do work that used to need a person or a hand-written script: drafting tests from requirements, writing and repairing automation code, running end-to-end flows against a live application, reading failures to find the cause, and spotting defects nobody wrote an assertion for.
The short version
Mature and boring: AI code review on the diff, and failure triage with logs, screenshots and recorded reasoning. Use both.
Good, with a catch: AI-written test scripts. Faster to author, exactly as fragile to maintain, and they tend to assert whatever the code currently does.
Real, with conditions: agentic end-to-end testing, where an agent runs a plain-language goal against the live app with no script. It isn't deterministic, it's slower per step, and it can route around a regression a selector would have caught.
Mostly marketing: "fully autonomous QA" with no human in the loop, and any pitch where "self-healing" carries the demo.
What decides success: test data and login before the trial, a flake policy before you gate a merge, and a named person who owns what "correct" means.
The rest of this piece is the detail behind those five lines, based on what we build and on 41 discovery calls with engineering and QA teams between June and early September 2026.
The problem AI is being asked to solve
Everyone says cost. Across those 41 calls, cost was the stated reason for looking on two of them. Pace was the reason on most of the rest, usually as some version of "we can't check things as fast as they now ship."
One media-analytics company moved its front end to almost entirely AI-generated code in three weeks, and the QA manager there found out roughly when we did. "The ways that we have been working as QA also needs to change to keep up." Elsewhere the numbers were less dramatic and pointed the same way: a QA function running a sprint or two behind development, a release that used to carry 15 pull requests carrying 200, the same people checking them. An engineering director at a games studio summarised where it lands: "the bottleneck for development is no longer development. Either you can't come up with enough good ideas to implement, or you can't verify that they're actually working."
Every one of those teams already had automation. The automation was the thing that couldn't keep up. When a Playwright suite needs an engineer's afternoon each time the design system moves, adding more Playwright is not the fix.
AI in testing is worth exactly as much as it closes the gap between how fast the code changes and how fast someone can say, with evidence, that it still works.
Where AI sits in the testing lifecycle
Most tool comparisons fail because two products called "AI testing" are doing different jobs at different points in the lifecycle. Here is where the work lands, in the order it happens.
Design: deciding what to test
Give a model a ticket or a PRD and it produces test cases. This is the oldest use and the most over-sold. Models are good at enumerating the obvious paths and mediocre at the paths that matter, because the paths that matter depend on knowing the product. An e-learning company trialled this in an earlier generation and kept the output for a quarter; their QA director's summary was that it generated tests but was "not really catching things we did not know already." It's a decent starting checklist. Teams keep the fraction that maps to flows the product has and discard the rest.
It gets better when the generator has seen the application, not just the ticket. An agent that has crawled the product can propose tests for the screens that exist, and can show you which screens have nothing at all. Goals drafted from a crawl describe the app as it is, bugs included, so use them to find what you forgot to specify rather than as the specification.
Author: producing the test
Two very different things happen here under one label.
AI-written scripts. Claude, Cursor, Codex and Playwright's own codegen will produce a test script from a description or a recording. This is good now. A QA lead in life sciences told us tools like this took test creation "from weeks of work to days of work" for his team. The catch is that the output is a script. Selectors, waits, fixtures. Faster to write, exactly as fragile to keep. You've moved the bottleneck from authoring to maintenance, which for most teams was already the bigger cost. The trade is laid out case by case in QA.tech vs Claude and Playwright.
A note on self-healing. Some script-based tools add a repair step: when a selector breaks, the tool guesses which element you meant and patches it. Maintenance drops. In exchange, your safety net now contains a guess nobody reviewed, and when the guess is wrong the test passes while checking the wrong thing. An agent doesn't remove that guess; it makes a fresh one every run, with the reasoning recorded so you can read it. Self-healing hides one guess in your repository; an agent puts a new one in the log each time. If your compliance regime needs a reviewable change record for the test itself, ask any agentic vendor three things: are goals versioned somewhere you control, is every run's full trace retained and exportable, and can you reconstruct in twelve months what was executed and why it passed. If a tool can't show all three, don't gate a regulated release on it. Our longer argument on self-healing is in the top 5 things everyone gets wrong about self-healing tests.
Goals for an agent. The other kind of authoring is a sentence: "a logged-out user can reset their password from the login page, follow the emailed link, set a new one, and log in with it." No selectors, no code. What you maintain instead is the goal, the test accounts and data the agent uses, and the constraints you give it. Where a judgement call isn't good enough, you pin an explicit check inside the goal, for instance that a specific request returned a specific status and body.
Run: executing the test and judging the result
AI code review reads the diff and comments on logic, style and risk. It's table stakes now. What it can't do is tell you whether the feature works for a user, because it never ran the product. A quality engineering lead at a collaboration-software company put the gap in one sentence: they had tools to check that AI-written code was good, "but we are not sure from the end user perspective, is it good enough."
Change-scoped testing on the pull request is the next step, and where we spend most of our time. Read the diff to decide which flows are affected, run them against the preview environment, post a verdict back before anyone merges. We call this dynamic testing. Static analysis and code review read the diff; this exercises the running product, and unlike a nightly suite it does so while the branch is still open and the author still has the change in their head.
Agentic end-to-end testing is the category most people mean when they say AI testing in 2026, and the one where the marketing runs furthest ahead of reality. An agentic test has no script. On each run the agent reads the live screen, decides the next step, takes it, and judges whether the outcome matches the goal. It drives a real browser or a mobile emulator. Most platforms keep a map of the application between runs so the agent isn't rediscovering the product every time, and that map can go stale the way any cache does; a good platform treats it as a hint to check against the live page, not a contract, and a moved button doesn't break the test.
That cuts both ways, and it's the trade to interrogate hardest. An agent that re-derives the path will sometimes route around a real regression and report a pass where a selector-based test would have gone red. Agentic suites buy a lower false-failure rate with a higher miss rate on a specific class of bug. Seed three known bugs into a branch, run every tool you're evaluating against it, and count what each one catches. Ask any vendor, us included, for a miss rate on that exercise, and treat a missing answer as an answer. If "the UI is unchanged" is what you need to assert, you want a visual baseline or a selector, not an agent.
QA.tech is an AI QA testing platform built this way: autonomous agents run plain-language goals against web and native mobile apps, on every pull request and on a schedule, with no scripts or selectors to maintain. What is AI QA testing goes through the mechanics step by step; what is agentic testing separates the term from its neighbours.
What agents do well is flows that change often, flows nobody had time to script, and the runtime judgement a script can't make, like a total that's arithmetically right and shown in the wrong currency. What they do less well is byte-identical repetition. The agent is a statistical model. Where a flow has two sensible paths it may take either, and there are flows where you should keep the script.
Diagnose: working out what broke
The unglamorous stage, and the one teams notice first. A failing agentic test arrives with the reasoning at the point of failure, the screenshot, the request that returned a 500, the console output. "Test 47 failed" becomes "the invite email never arrived and here's the request that didn't fire." A head of QA who ran two agentic tools side by side described the difference between them as being able to read a failure "at console level, log level, UI level, and image level" rather than a pass/fail line.
What's real and what isn't
A scorecard, from what we've seen work on real products this year.
Real and mature: AI code review on the diff. Failure triage with logs, screenshots and recorded reasoning.
Real, with a caveat: AI-written unit and integration tests inside a coding agent. They're fast and they run. They also tend to assert whatever the code currently does, which means they lock in the bug as readily as the behaviour, and coverage climbs without escaped defects falling. Read every assertion, or write them yourself. Automated accessibility scanning is in the same bucket: it reliably catches the subset of WCAG failures that are machine-detectable, and a clean scan is not an accessible product.
Real, with conditions: Agentic end-to-end testing on web and native mobile. It works. It isn't deterministic, it's slower per step than a script, it can miss the regression it routes around, and it needs someone who knows what the product should do to write the goals.
Marketing, mostly: "Fully autonomous QA" with no human in the loop. Test generation that finds bugs your team didn't know about from a ticket alone. Any pitch where the words "self-healing" carry the whole demo.
Barely touched: performance, load and security testing. AI helps with analysis and scripting at the edges; the discipline itself hasn't changed much yet. Service-level testing sits in between: API calls can run as steps inside an agentic flow, but standalone API regression is still script territory.
The scepticism is earned. A quality director at a financial-services firm had tried three AI testing products and all of them failed on his apps: "I'm quite a cynic in this space." A QA lead in life sciences had "looked at some other tools that are what I believe are similar", and found "they haven't really been as all-encompassing as they make it seem in the marketing material."
Model drift, and pinning what runs
One objection almost nobody writes about. An agent passes today; the model provider ships a new version next month and the same goal behaves differently. Ask any vendor two things: do you choose which agent runs your suite, and what happens underneath it when providers retire models. In QA.tech you pick the agent per test (a fast one for stable flows like login, a more capable one for mobile and complex flows), and we swap the models underneath a tier as providers change them, after testing. That's the honest shape for every vendor in this category, because nobody controls the model providers' release calendar. What you can insist on is being told when it happens, and having your regression suite run once before and once after.
Test data and environments, the part every pilot trips on
Every end-to-end programme dies in the same place, and it's not the tests. It's the data.
An agent that can sign up, check out and cancel a subscription is worthless if it has to do it against a staging database three other people are also mutating, or against a seeded account whose trial expired last Tuesday. Before you evaluate a single tool, answer three questions. Where does a clean test account come from? Who resets it? What happens when a run leaves a half-finished order behind?
Login is the other half. If your app has SSO, an OTP or a WebAuthn prompt, an agent can't reason its way through it any more than a script can. In practice you give the platform seeded accounts on a non-production environment, a mailbox it can read for magic links and codes, and a reset path. AI helps at the edges here (models are decent at generating plausible synthetic records and at spotting PII you forgot to mask), but it doesn't solve environment isolation, and any vendor who implies it does hasn't run against your stack. Budget half a day with your platform team before the trial.
Flakiness, and how you gate a merge on a non-deterministic test
The obvious follow-up to "agents aren't deterministic" is the one every engineering lead asks in the first ten minutes: how do I block a merge on something that might do two different things?
You treat agentic runs the way mature teams already treat any end-to-end suite. Measure the flake rate per flow over at least 30 runs before you gate anything on it; a fortnight of nightlies is 14 samples, which can't tell a 2% flake rate from a 10% one, and that difference is the whole gating decision. Quarantine flows that fail intermittently rather than deleting them or retrying until green. Distinguish "the agent took a different route and still got there" from "the agent couldn't get there", because only the second is a build breaker, and a good platform's evidence makes the two easy to tell apart. And publish the false-failure rate to the developers you're asking to trust it, because they'll compute it themselves otherwise.
One more constraint: wall-clock time. In our experience a pull-request check that takes longer than about ten minutes gets merged past, and then it's a nightly with extra steps. Scope the PR run to the affected flows and keep the full pass for the schedule.
What it costs
No one publishes list pricing, and we're not going to here either, because it depends on flow length. What we can give you is the model.
Agentic runs cost real inference money. The unit is work done, not tests owned: QA.tech bills per test-case execution, others per step or per token. Two variables move the bill. Memory, because a run on an app the platform already knows spends its steps on the flow you asked about, while a cold run spends them finding out where things are. And scope, because a run limited to the flows a diff touches costs a fraction of a full suite. "Run everything on every PR" is a budget decision, not a config setting. Ask whether a re-run after a flake bills again.
Model it against your own number, not ours. Track what your team spends on test maintenance for one month, then compare. The range across the teams we've seen is wide enough that any default we gave you would be wrong for you. Expect the first quarter to cost more than the steady state, because you're running both systems while you learn which flows to move.
Security
Every fintech, life-sciences and gambling team we talk to asks where the screenshots go before they ask anything else. An agent driving your staging environment is sending page content and screenshots to a model provider at inference time. Ask which providers, in which regions, under what terms, and whether anything trains on it. For QA.tech: SOC 2 Type 2, data at rest in the EU, model providers under zero-retention and no-training terms, listed with the sub-processors in the trust centre. Our terms also require synthetic test data. That's a constraint on you rather than a feature of ours, and if your staging environment carries production-shaped records it's a conversation to have before the trial rather than after.
How teams roll it out
Pick the flows you'd roll back a release for. A head of QA at a gambling operator described his pilot this way: "we're not focusing on creating the 500 test cases for regression, but we're looking at the 10 core journeys within this application." Sign-up, login, the thing customers pay for, the thing that generates the invoice. Run those first.
Put it on the pull request, not just the nightly. A nightly failure lands in a Slack channel the morning after the author has moved on. A PR failure lands while the branch is still open.
Keep your scripts where they earn their place. Nobody should throw away a working Playwright suite. Move the flows that keep breaking and the flows you never covered. Keep the stable, precise checks where determinism matters. Our recommended shape is a small scheduled regression suite of must-never-break flows, plus dynamic testing on every pull request scoped to what changed. Run both suites side by side for a quarter, retire scripts one flow at a time as the agentic version proves itself on the flake and miss numbers above, and expect to keep some scripts permanently.
Decide who owns quality before you buy the tool. A CTO at a marketplace company declined to adopt an agent because he worried about "cognitive surrender", his team blaming the tool for misses instead of owning them. The tool takes the repetitive pass and the waiting. The judgement about what "right" means stays with a person, and it should be a named person.
Measure the thing you were short of. Not test count. Time from "ready for test" to verdict. Escaped defects per release. Hours your engineers spend on test maintenance.
What this means for QA engineers
The job changes shape before it changes size. In the teams we've watched go through this, the day-to-day was different within a quarter and the headcount conversation came much later, if at all.
The repetitive regression pass goes to the agents. So does the queue: one streaming company had tickets sitting 17 days in a "ready for QA" column before anyone opened them, on a team with most of its regression already scripted, and its QA lead described the team as "in a burnout position." What's left is the work that needed a person all along: deciding what to test, judging what "correct" means for this product, exploratory testing, and being the one who reads the evidence and makes the call. It's a smaller team doing a more senior job, and a different job from the one most QA engineers were hired for. We've written about the skills shift in restructuring your QA team for the AI era.
Writing the tests turned out to be the easy half. Deciding whether the result is good enough to ship still lands on a named person, and no credible vendor is claiming otherwise.
If you want to see agent-run testing against your own product rather than a slide, the QA.tech AI testing platform runs a first test in a few minutes, and a 30-minute demo covers the rest.
Common questions
What is AI in software testing? Using AI models and agents to draft tests, write or repair automation code, run end-to-end flows against a live application, detect defects nobody asserted for, and diagnose failures. It ranges from AI-assisted scripting, which is mature, to agentic testing with no scripts at all, which works with conditions.
How is AI used in software testing today? Across four stages: designing what to test, authoring the test (as a script or as a goal for an agent), running it and judging the result, and diagnosing failures. AI code review and failure triage are the most widely adopted; agentic execution is the newest.
Will AI replace software testers? It replaces the repetitive regression pass and the waiting in the queue. It doesn't replace the person who knows what the product should do, writes the goals, and reads the evidence. Most teams end up with a smaller QA function doing a more senior job.
What is the difference between AI testing and AI test automation? AI test automation usually means AI helping to write or maintain scripts that then run conventionally. AI testing, in the agentic sense, means an agent running the test itself from a goal, with no script. The second is the newer category and the one with the bigger change to maintenance.
Is AI testing reliable? It depends which kind. AI-written scripts are as reliable as any script. Agentic tests are not deterministic: where a flow has two sensible paths the agent may take either, and an agent can route around a regression a selector would have caught. Reliable tools bound the run, record evidence, tell you when the model underneath changes, and give you a verdict you can audit.
How do I start using AI in software testing? Pick the flows you'd roll back a release for, run them with an agent against a preview environment on every pull request, keep your existing scripts where they're stable, sort out test accounts and data before the trial, and measure time-to-verdict and escaped defects rather than test count.