Contents
AI QA testing means software tests that are planned, run and checked by an AI agent working from a goal you wrote in plain language, rather than a script someone coded step by step. The agent reads the screen, decides what to do next, and judges whether the outcome matches what you asked for. When the interface changes, it works out a new path instead of failing on a selector. That is the appeal and also the risk, because an agent will route around a bug as readily as around a redesign.
The short version
Three names, one term
AI-assisted scripting (a model writes Playwright for you), self-healing scripts (a tool patches selectors when they break), and agentic testing (no script; an agent re-derives the steps from a goal on every run). Only the third removes selector maintenance. This guide is about the third.
What changes
Authoring becomes a sentence, the selector tax goes away, and "what are we not testing" gets an answer for the first time.
What doesn't
It isn't deterministic, it can route around a regression a selector would have caught, it only tests the goal you gave it, and login and test data stop more pilots than anything else.
What it costs
Work done, not tests owned. Runs are slower per step than scripts and parallelise, so scope pull-request runs to what changed and keep the full pass for the schedule.
What to ask a vendor
How runs are bounded, what happens when the app changes incorrectly, whether it needs your code, where the screenshots go, and what you're paying for on top of the model.
QA.tech builds autonomous QA agents of exactly this kind, so read what follows knowing who wrote it. It's written for the engineering lead or QA lead who has already decided AI should be doing more of the testing and now has to pick an approach, and it spends as much time on where this doesn't work as on where it does.
What "AI QA testing" covers, and what it doesn't
Vendors have an interest in blurring three things that behave completely differently in CI.
AI-assisted scripting. A model writes Playwright or Cypress code for you, from a prompt or a recording. The test that runs in CI is still a script with selectors. Faster to author, the same to maintain. Claude with the Playwright MCP, Cursor, and most "AI test generation" features live here.
Self-healing scripts. Still a script, but when a selector breaks the tool guesses which element you meant and patches it. Less maintenance, and a guess in your safety net you may never review.
Agentic, or autonomous, testing. No script. The test is a goal ("a new customer can sign up, add a card and get charged the right amount"), and an agent re-derives the steps on every run against the app as it exists that day. This is what QA.tech does, and it's what this guide means by AI QA testing unless it says otherwise. The two terms get pulled apart in what agentic testing actually is and in what autonomous testing is.
There's a fourth confusion worth clearing up because it trips up search results and sales calls alike. AI testing is not the same as testing AI. Evaluating a model's outputs or red-teaming a chatbot is a different discipline. This guide is about using AI to test ordinary software. Some of that software will contain AI features, and agents can test those too, but the subject here is the tester.
Why teams are looking at this now
Nobody opens these calls by telling us AI is exciting. They open with a number that stopped making sense: releases that used to carry 15 pull requests now carry 150 to 200, on the same QA headcount.
"I think we're rolling a dice. A sketchier dice every time I go to production."
Automating most of a regression suite doesn't help if the suite still runs as one batch before release. Tickets queue in "ready for test" behind a manual pass nobody can parallelise, and we have seen that queue reach 17 days on a team with four fifths of its regression already scripted. The wait is a scheduling problem, and more scripts don't touch it.
Then there are the teams with no QA function at all. Engineers test their own changes locally, the product owner is the last gate, and things slip through into support tickets. For them the question isn't how to speed QA up. It's how to have any.
Both groups ship more code than they can vouch for, which is what to measure a tool against. Pick three numbers before you start a trial and read them again at 90 days: escaped defects per release, hours between a pull request opening and someone knowing whether it broke something, and the share of your critical flows covered by anything at all.
How an AI QA agent actually runs a test
Five steps. The awkward questions live in the third and fourth.
1. It maps the application. Before writing anything, the agent crawls the product: routes, screens, forms, the actions available on each. QA.tech stores this as a map of pages, elements and actions that persists between runs, so the second run starts knowing what the first learned, and the agent updates it as later runs find changes. This is also where coverage discovery comes from, since a mapped app can tell you which of its flows have never been exercised. It can't tell you about the flow that was never built. More on that below.
2. You describe what should be true. In plain language, in a chat, or generated from a ticket or a pull request. "An admin can invite a user, the user receives the email, accepts, and lands on the dashboard with the right role." No selectors, no code. Domain rules you add ("invoices are always in SEK for Swedish accounts") stick and apply to every later run, and you can pin explicit checks inside a goal, such as a specific request returning a specific status and body, where a judgement call isn't good enough. The crawl can draft goals too, and this is the one place to be careful. A goal derived from the app describes the app, bugs included. Treat crawl-drafted goals as a checklist of things you forgot to specify.
3. It plans and acts in the real UI. On each run the agent looks at the live screen, decides the next step towards the goal, takes it, and looks again. It drives Chromium for web, and the iOS Simulator and Android emulators for native mobile. If a button moved or was renamed since last week, nothing referred to the old one, so there's nothing to break; the agent looks at the screen it has and finds a route. Two details decide whether this step is reliable, and you should ask any vendor about both. How "click the submit button" becomes an action on a specific element: resolving to an accessibility node or DOM handle is far more stable than predicting screen coordinates from a screenshot. And how the agent knows the page has settled before it looks again, because an agent that acts on a half-rendered page produces exactly the flakiness you adopted it to escape.
4. It judges the outcome. A script asserts exactly what someone thought to assert when they wrote it. You can bolt generic checks onto that, and you should: axe-core for accessibility, a console listener that fails the run on an uncaught error. Those are still rules a person wrote in advance. The agent evaluates whether the goal was met in the terms you set, which catches the class of failure nobody writes a rule for, like a total that's arithmetically right and formatted in the wrong currency, or a confirmation screen that loads with the previous customer's name still in the header. The catch is that the judge is a model reading a screen and a log and writing down what it thinks happened. It can be confidently wrong in exactly the way a hurried human reviewer is, in both directions.
5. It leaves evidence. The verdict comes with the agent's reasoning at the point of failure, alongside the screenshot, video and network log a Playwright trace would also give you. The reasoning is what's different. "Test 47 failed" turns into "the invite email never arrived; here's the request that didn't fire", and someone who didn't write the goal can open a red run and decide in a couple of minutes whether it's a bug or the agent being wrong.
Agentic QA vs script-based regression testing
What changes for a team
Authoring stops being a development project. Writing a Playwright suite for a product of any size is weeks of engineering that competes with the roadmap. A goal is a sentence, and the people who can write one include the product manager and the support lead, who between them know more about what the product should do than anyone on the QA team.
Maintenance changes shape rather than disappearing. The selector tax goes away because there are no selectors, and that work was proportional to how often a class name changed rather than how often the product did. Two costs replace it. Real change to the flow: if login gains a second factor the test never knew about, someone updates the goal. And verdict review: a model judging an outcome will sometimes fail a run that was fine, or pass one that wasn't, and somebody looks at the evidence and decides. Teams plan for the authoring time they're saving and not for the review time they're taking on. Budget for it.
Coverage becomes a discovery problem instead of an authoring one. Because the agent has mapped the app, "which of these flows have we never exercised" has an answer for the first time. The harder gap, the flow that was never built, stays yours.
Regression suites can stay small. Our recommended shape, and it's the pairing that keeps coming back in customer conversations: a short scheduled regression suite of flows that must never break, plus dynamic testing scoped to what actually changed on every pull request. Not the 1,300-case pack that runs every fortnight because nobody wants to be the person who deleted a test and was wrong. What regression testing is for has the longer argument.
What doesn't change, or gets worse
It isn't deterministic, in two ways. The execution agents are statistical models. Where a flow has one sensible path the agent takes it consistently; where it has two, it may take either. That's path variance, and it's mostly harmless. Verdict variance is the one to care about: the same goal against the same build can occasionally get a different judgement. A Playwright script will click the same pixel every time until it can't. We don't publish a false-fail rate yet, because we haven't measured one across enough stable suites to stand behind it. What we can tell you from pilots is where the noise comes from: almost every early false fail is the agent lacking context (it didn't know a pop-up was expected, or a test-environment label was fine), and it stops once that's written into the goal or a rule. Ask us to measure it on your suite during a pilot; that number is worth more than an average.
It can adapt around a regression. The flip side of "a moved button doesn't break it". If the button moved because someone shipped a bug, an agent will often find it anyway and pass, where a selector-based test would have gone red. Agentic tests check that the user's goal is still achievable. They don't check that the UI is unchanged. If unchanged is what you need to assert, you want a visual baseline or a script. The practical test during an evaluation: seed three known bugs into a branch, run every tool you're considering against it, and count what each one catches.
The agent tests the goal you gave it. If the application is built wrong and the goal was derived from the application, the goal is wrong too. Goals have to come from what the product should do, which is tickets, acceptance criteria and a human's understanding, not from whatever the app happens to do today. QA.tech reads Linear and Jira issues for exactly this reason, and it still needs someone who knows what "right" means.
It won't find what isn't there. A team that went a different way after a trial made a fair point: the regressions that hurt them most were "things that weren't being built", logical omissions rather than broken flows. An agent exploring the app can't notice a feature that doesn't exist. Neither can a script.
Chromium, and simulators. Web runs on Chromium. Native mobile runs on the iOS Simulator and Android emulators, not a real-device farm, so bugs that depend on real hardware, real network conditions or a specific older handset aren't covered, and neither is Safari. If your bugs live there, you still want a device lab underneath and this is not a replacement for it.
Login and test data stop pilots more often than anything else. If your app has SSO, an OTP or a WebAuthn prompt, an agent can't reason its way through it any more than a script can, and a vendor implying otherwise is describing a demo account. In practice you give the platform seeded test accounts on a non-production environment, a mailbox it can read for magic links and codes, and a way to reset state between runs. Budget half a day for this with your platform team before the trial starts.
What a run costs, and how long it takes
A scripted login takes a few seconds. An agent reasoning through the same flow takes closer to a minute, because it looks at the screen between every action, and a second agent then reads the whole trace to decide the verdict. Agent runs parallelise, so a 40-flow pull-request check is a concurrency question more than a latency one. Ask any vendor what the cap is and why. Ours: there's no ceiling on how many tests a run can contain, but parallel executions are capped per environment by default so the agents don't overload your staging, and the cap is configurable. Either way, agentic testing is a poor fit for a pre-commit hook and a good fit for a pull-request check.
On cost, the unit is work done. QA.tech bills per test-case execution; other vendors bill per step or per token. Whichever it is, two things move the number. Memory, because a run on a known app spends its steps on the flow you asked about while a cold run spends them finding out where things are. And scope, because a run limited to the flows a diff touches costs a fraction of a full suite. Ask any vendor for the step count of the same goal on run one and run ten, whether the bill moves when a flow gets longer, and whether a re-run after a flake bills again. Expect the first quarter to cost more than the steady state, because you're running both systems while you learn which flows to move.
Which flows to keep as scripts
Keep the script for third-party payment redirects where the provider's sandbox UI is fixed and you need the same path every time; pixel-exact visual diffs; canvas, WebGL and map interactions an agent reads poorly; anything you run a thousand times as a load or soak test; and any check a regulator wants as a byte-identical audit each quarter.
Put agents on multi-step business flows, anything with a lot of conditional state, anything that changes weekly, native mobile flows you never scripted per platform, and the long tail nobody ever got round to covering.
If a flow is stable, cheap to script and business-critical, a script is still the cheaper answer. The full comparison of QA.tech against Claude and Playwright goes through the cases one by one.
Questions to ask any AI QA testing vendor
Is it deterministic, and how do you handle a run that goes off the rails? A CTO who had experimented with unbounded agents described them "solving each other's made-up problems." A serious vendor should tell you how runs are bounded, how a verdict is reached, and what evidence you get to check it. Ours: a test case is capped at around 50 agent steps, and one that runs out of budget stops and reports rather than wandering. That's deliberate. A 50-step test is a smell; the teams that hit the cap were adding context to a single sprawling case instead of splitting it, and short cases with a prerequisite ("test case 12 has completed") are more reliable and cheaper to read. A pull-request run whose tests never actually exercised the change comes back "unverified" rather than green.
What happens when the app changes incorrectly? Adapting to change is the pitch. So what stops the agent adapting around a bug? The answer should involve the goal being independent of the UI, and a human-readable verdict you can audit.
Does it need access to our code? Two separate grants, and treat them as separate decisions. The runtime reads no source: tests run against the UI and the network layer, and you can run the whole product that way. The pull-request integration installs a GitHub or GitLab app with read access to the repository so it can see the diff, and your security team will correctly classify that as source access. It uses the diff to decide which flows to run, never as the source of truth for what the app should do. Turn it off and you lose change-scoped testing, not the platform.
Where does the data go? An agent driving your staging environment is sending screenshots and page content to a model provider at inference time. At-rest residency isn't the question; ask which providers, in which regions, under what terms. Ours: SOC 2 Type 2, data at rest in the EU, model providers under zero-retention and no-training terms, all listed with the sub-processors in the trust centre. One thing to check before you sign rather than after: our terms require synthetic test data, so if your staging environment runs on a scrubbed production snapshot, that's a conversation for week one.
How deep does it go into CI/CD? Does it comment on pull requests, is there a webhook, is there an MCP server, how do you interact with it day to day? A tool that lives in its own dashboard is a tool nobody opens. QA.tech posts verdicts back to GitHub and GitLab, runs from GitHub Actions, the CLI and the REST API, and has an MCP server so Claude Code, Cursor or Codex can run and inspect tests from the editor.
Can we take our tests with us? This question has cost us a deal, so here is the answer in full. QA.tech tests are goals executed by an agent, not scripts compiled to code, so there is nothing to export as a runnable Playwright file. You can export every test definition with its steps through the API or CLI as a readable record, and keep that in git if you want a change history you control. One CTO told us he considers his tests "as much IP as the product itself." If that's your position, the goal-based category may not be where you should buy.
What am I paying for on top of the model? The other question that has cost us deals, usually to a team that decided to wire Claude to Playwright themselves. The answer should be specific: the application map, the runtime judgement, the evidence, the integrations, the parallel infrastructure, the person you call when a verdict looks wrong. If a vendor can't articulate what sits between you and the raw model, you can probably build it yourself, and some teams do.
Can it tell us what we're not testing? Most tools report on the tests you have. Ask to see the gap.
If self-healing is the headline feature, ask to see the repair log. A tool that heals often has tests that break often. The useful questions are what last month's repairs looked like, how many a human reviewed, and what happened the time it healed a test into passing against a bug. Our position on the whole idea is in the top 5 things everyone gets wrong about self-healing tests.
Does AI QA testing replace Playwright?
Usually it sits next to it. Keep scripts where byte-identical repetition matters and the flow is stable. Put agents on everything that changes, everything that's expensive to script, and everything you never got round to covering. The section on which flows to keep as scripts is the practical version.
Does AI QA testing replace QA engineers?
It replaces the repetitive regression pass and the waiting in the queue. It doesn't replace the person who knows what the product should do, writes the goals, and reads the evidence, and it adds a review job that didn't exist before. In the teams we've watched go through this, the day-to-day changed within a quarter and the headcount conversation came much later, if at all.
If you want to see the AI QA testing platform run against your own application, the fastest way is a 30-minute demo.