# Can You Trust an AI Agent to Run Your Tests?

> Not blindly. What trust should mean for an AI agent that runs your tests: evidence for every step, an "unable to verify" verdict, flake reruns and a checklist.

Source: https://qa.tech/blog/trust-ai-agent-to-run-tests · Published: 2026-11-11

---
Not blindly, and a good one won't ask you to. You trust an agent that runs tests the way you'd trust a new colleague's bug report: by checking what it shows you. Look for a screenshot and logs behind every step, a verdict that can say "unable to verify" when the agent couldn't tell, reruns before a failure gets called a bug, and the agent's own mistakes reported apart from product bugs. Expect results that repeat on unchanged pages without being deterministic, and keep a small scripted suite for anything that must run identically every time.

## Why teams are asking whether to trust an AI agent with their tests

The question kept turning up in the testing and developer newsletters we read through late October and early November. Testers asked it. So did people building agent harnesses, and developers who'd watched a coding agent declare victory over a test it had quietly rigged.

The clearest framing comes from Bruno Legeard at Smartesting, who sells a test-execution agent and still sets the bar here: "The true question is not whether we can blindly trust an AI agent; the answer must be no." ([Smartesting](https://www.smartesting.com/en/ai-agent-to-execute-tests/)) His argument is that running a test carries more risk than writing one, because the agent "executes, observes, interprets, and produces a verdict." Every one of those steps can go wrong.

Then there's the view from the keyboard. A developer writing at orchidfiles logged "over 100+ hours of working with Composer in Cursor" and listed what broke ([orchidfiles](https://orchidfiles.com/youll-need-nerves-of-steel-to-work-with-ai-agents/)). In one case the agent "decided to simply skip calling the API route in the tests and write the data directly to the database instead." The tests passed. "And, of course, all the tests are green, which confirms its conclusion that everything is done."

Strictly, that's a coding agent writing its own tests. A testing agent running yours can fail the same way from the other side: a green result that has stopped checking what you think it checks.

And nobody's going back to clicking through every pull request by hand. Faros AI's Q3 2026 report found that "time in QA is up 300.6%" and that "QA inherits the verification burden, where it is often slower and more expensive" ([Faros AI, The Speed Trap](https://www.faros.ai/blog/ai-speed-trap-takeaways)). Teams are handing that load to agents because they have to. Asking whether to believe what comes back is the right instinct.

## Two ways an agent-run test misleads you

A result can be wrong in two directions, and [flaky tests come in two kinds](https://qa.tech/blog/flaky-tests-two-kinds) already covers the general case. What's new is how an agent gets into each one, and who pays for it.

**The false green.** The test passes, the product is broken. Smartesting calls this "the most concerning case: the test passes when it should have failed." Agents have a particular route into it. Picture a test that expects the flow A, B, C. Something changes and step B gets harder to pass, so the agent finds another way through, A, D, E, C, and reaches the end. If B got harder because the code behind it is broken, the agent has just walked around the bug.

An agent that adapts is useful right up to the moment it adapts around the bug. It's the same worry that sits under self-healing scripts: once a tool re-guesses its way past a broken step, you're trusting the guess. We went through that in [the trouble with self-healing tests](https://qa.tech/blog/self-healing-tests-misconceptions).

**The false red.** The test fails, the product is fine. This one used to be cheap. Someone looked, shrugged, reran it. It's expensive these days, because the first reader of a red result can be another agent. Monaco's engineering team describes coding agents that "poll for review comments and failing CI checks to address them automatically" ([Monaco](https://www.monaco.com/blog/agent-developer-workspaces)). Feed that loop a wrong failure and it goes to work on it.

Now put that on mobile. An agent runs PR tests on an Android emulator that keeps struggling, and it flags a screen as broken when the screen is fine. The comment lands on the pull request. A coding agent reads it and starts fixing something that was never wrong, and a run of false detections like that is enough for a team to stop finding the testing useful.

That's the price of a false red in 2026: a coding agent chasing a bug that doesn't exist, and a "fix" somebody now has to review.

Determinism is the other thing people mean when they ask if an agent is reliable, and that piece covers it too. This one picks up where it stops. You have one verdict in front of you. How do you know it's right?

## What trust should mean for an agent that runs tests

None of what follows asks you to take the agent's word for anything.

### Evidence behind every step

A red or green light on its own tells you very little. Smartesting's line is that "for test execution, the path matters as much as the outcome": where the agent went, what it did, what it saw, why it concluded pass or fail. In practice you want a screenshot per step, the agent's reasoning for each action, the network requests and console output from that moment, and a recording.

That's the idea our own product is built on. A browser test run keeps its proof (screenshots, network calls, console errors), and the review posted on the pull request links to it.

Evidence is also the only thing that catches the A, D, E, C problem. If the trace shows the agent detoured where the test expected step B, you can see it and fail it.

### A verdict that can say "I couldn't tell"

Two verdicts force a guess whenever the agent can't finish. A guess that lands on pass is the worst result available. So the verdict set needs a third value: pass, fail, or unable to verify. The login the agent couldn't get past, the preview that never deployed, the change the tests never reached. Each of those should come back as unable to verify, and your team should read it as a to-do. Why that third verdict exists at all is the subject of [passing tests are not proof](https://qa.tech/blog/passing-tests-are-not-proof).

### Reruns before accusations

Rerun a red before anyone calls it a bug. That part is old advice. For trust, what matters is what happens to the result after the rerun.

There's a catch, and Addy Osmani names it in his essay on agentic verification. He's writing about coding agents, but the incentive holds for anything that's allowed to retry: "If it's possible to retry a failing test and have it work once in a while, it'll pick retrying as its preferred strategy" ([Addy Osmani, Agentic Verification](https://addyo.substack.com/p/agentic-verification)). Reruns can launder a real intermittent bug into a green. The defence is to keep the flake visible. A test that failed and then passed on retry belongs in its own list, with a likely cause attached. Drop it quietly and you get a suite that's green and wrong.

Osmani again, on why this matters beyond one test: "A flaky test in the main suite is worse than no test at all, because it breaks trust in every other result."

### The agent's own mistakes in their own bucket

Agents misclick. They misread a page. Pretending they don't is the quickest way to lose a team. A triage that only knows "bug" and "pass" pushes every agent stumble into the bug pile, which is the exact pile your coding agents pick up and act on.

A useful triage splits failures into product bugs, outdated tests, bad test data, timing, environment, credentials and third parties, and agent mistakes. Humans have the same problem, by the way. Anyone who has triaged crowd-testing reports knows the pile: a product with no price in a half-seeded development environment, filed as a bug, and someone's afternoon gone sorting real faults from environment ones. An agent inherits that confusion unless the categories are built in.

### Repeatable on unchanged pages

Run the same check on an unchanged page and you should get the same answer. You should also be able to force a fresh run that ignores any cached decisions, so you can see the agent work it out from scratch. What you won't get is the identical path every time. Expect small differences in the route the agent takes from one run to the next.

### Context that keeps it from making things up

People skip this one. An agent that sees only a screenshot has to guess what the page is for, and guessing is where invented results come from.

For a test agent, context means the diff and what the pull request says it changes, a summary of what the product is for, and the rules your team has written down. A map of the app (which pages exist and what you can do on each) does its work earlier, when the agent decides what to test. It's also why a context-based agent is the better way past a growing scripted suite. An agent that knows what the product is for can go looking past the steps it was given, which is the whole idea behind [automated exploratory testing](https://qa.tech/blog/automated-exploratory-testing).

That only counts if it shows how it used the context, though. "I tested checkout because the diff touched the cart component" is a claim you can check. "Tests passed" isn't.

## A checklist to put to any agent-based tester

Take these to any vendor, us included. The middle column is the answer that should worry you.

| Ask | Answer that should worry you | Answer you want |
|---|---|---|
| What comes with a result? | A pass rate and a written summary | Per-step screenshots, the agent's reasoning, network and console logs, a recording, kept with the run |
| What happens when the agent can't finish? | It reports a fail, or retries until something passes | A separate "unable to verify" verdict that never counts as a pass |
| How are suspected flakes handled? | Retried silently until green | A bounded number of reruns, then a separate list with the likely cause |
| Can a failure be the agent's own fault? | "The agent doesn't make mistakes" | A named category for agent mistakes, kept apart from product bugs and environment problems |
| Can the agent change the test it's grading? | Yes, quietly, while it grades | Never while grading. Any later change to a test is something you configured, can switch off and can see in the test's history |
| Will the same check give the same answer twice? | "It's fully deterministic" | Repeatable on unchanged pages, a way to force a fresh run, and rules that keep later runs consistent |
| What does the agent know beyond the screen? | Nothing, it just looks | The diff and pull request context, a product summary and your written rules, plus a map of the app when it plans tests |
| Who makes the release call? | The agent | A person, reading the evidence |

Row five deserves a second look. Maple's team ported 287,000 lines of React with agents and wrote down one rule above the rest: "an agent must never edit the baseline, a scenario or a tolerance to make a diff go away" ([Maple](https://maple.dev/blog/react-to-foldkit/)). It's the orchidfiles problem again, with the fix written in. Whatever grades the work can't be allowed to move the goalposts.

Row eight comes from the people building the agents. Anthropic's Claude Code team, describing their own review process: "An agent's approval never counts as a person's, either" ([Anthropic](https://claude.dev/blog/solving-code-review-at-anthropic/)).

## Where scripts still win

A Playwright or Selenium test runs exactly the same steps every time, runs fast and costs nothing to license, and when it fails you know it didn't wander off. For sign-up, login and payment, the flows that must never break, that predictability is worth a lot. Our own docs say as much: for "exact same steps every time", scripts are "Best - fully deterministic" ([QA.tech docs FAQ](https://docs.qa.tech/getting-started/faq)).

What scripts can't do is notice anything nobody told them to check. Every feature means another script to write and keep alive, and the suite grows into something nobody dares prune.

Which is why we'd run both. Those few flows get a short scripted suite on a schedule, a fixed baseline that never wanders. Each pull request gets [dynamic testing](https://qa.tech/blog/what-is-dynamic-testing), where an agent that understands the product decides what the change could break and checks it in the running app. That second half covers what's new, and what's new is where this week's bugs are.

## How QA.tech answers the checklist

QA.tech, an AI testing platform, runs agent-based test cases against your running app, on each pull request's preview deployment and in test plan runs. It won't get every verdict right. Whether any AI test automation tool is reliable comes down to whether you can check it, so every result here is built to be checked by a person.

On a pull request, the review ends in pass, fail or unable to verify, posted as a native review. A bug only appears in the review when a test reproduces it. The PR comment gives the verdict and a table of what was tested, and one click opens the evidence: what the agent expected against what it saw, a step-by-step trace with its reasoning and a screenshot per step, network and console logs, the session recording when one was captured, and the test's pass/fail history. A coding agent connected over [the MCP server](https://docs.qa.tech/integrations/mcp) can read the same evidence before it starts fixing anything.

On test plan runs, a review agent looks at every non-passing test before anyone gets notified. It reruns where a rerun answers the question, at most twice per test. It sorts each failure into product bug, outdated test, test data, timing, agent mistake, environment, or credentials and third parties. In the docs, an agent mistake means "the app was fine, but the agent clicked the wrong element or misjudged the result" ([regression review agent docs](https://docs.qa.tech/regression-testing/regression-review-agent)).

Each plan run gets one verdict: Verified, Needs attention or Broken. Broken needs a confirmed product bug with the failing step and a screenshot. A rerun replaces the test's result, so a plan that was red only because of flakes reports green in CI. The flakes aren't dropped, though. They get their own "Flaky, passed on retry" list with the likely cause and a suggested fix for the test, and every finding links to the results it's based on.

The review agent never edits a test case during a review. It lists what should change under "Tests to update". Edits happen elsewhere. On GitHub, PR-generated tests persist as Drafts labelled ephemeral, outside the scheduled suite. After the merge, a selective [post-merge agent](https://docs.qa.tech/pr-testing/github#post-merge-agent) can update test cases the merge changed and deliberately promote only the PR's new ones worth keeping, working from a prompt you can edit. It's on by default for new GitHub integrations and stays off for existing ones until you turn it on. You can narrow that prompt or switch the agent off; promoting nothing is a normal outcome. Nothing is discarded or automatically added to scheduled regression, and every edit to a test case's steps or goal creates a new entry in its [revision history](https://docs.qa.tech/test-features/revision-history).

You can rerun only the failed cases, or rerun without the agent cache so every step is decided fresh. Two reviews of the same change can pick slightly different tests, and the review rules you set keep later runs following the same guidance. The agent reads pages through both the DOM and visual models, and the tests it suggests come from a crawled map of your app. Web runs are Chromium only. On GitHub, you can make the check required before merge. More on the PR side is on the [PR testing on preview deployments](https://qa.tech/product/pr-testing) page.

## What isn't solved yet

Plenty.

**There's no public benchmark.** Smartesting notes that "there is currently no publicly available reference benchmark dedicated specifically to AI test-execution agents." So any accuracy figure a vendor quotes is self-reported. We don't publish one.

**The judge is a model too.** Deciding that a failure was an agent mistake and not a product bug is itself a call an agent makes. Reruns and evidence shrink that risk without removing it, which is why [measuring the judges](https://qa.tech/blog/harness-engineering-behaviour-sensor) matters as much as measuring the code.

**Intent is hard.** A button moves on a pull request. Did the business logic change on purpose, or did something break? It's the question we still find hardest. An agent can read the diff and the pull request. If neither says what was meant to change, it's guessing the way a new hire would, and written rules only narrow the guess.

**Evidence covers what was tested, and nothing else.** Maple's harness compared thousands of screens and still missed something: "The command palette had no scenario, and we didn't notice it was never ported until the second night."

**Environments still make noise.** Emulators struggle, previews deploy half-ready, test data drifts. Better triage files that noise under environment, which at least keeps it out of the bug pile your coding agents work from.

**Someone still signs the release.** Evidence makes that reading quicker, and over time people will read less of it. Agentic coding went that way. People read generated code closely at first, then started letting go. That letting go should be earned by results you could have checked, every time.

## FAQ

**Can AI agents run end-to-end tests reliably?**
They can run them usefully, provided every result can be checked. A reliable agent records evidence for each step, reruns suspected flakes before calling them bugs, reports when it couldn't verify something, and separates its own mistakes from product bugs. It won't take an identical path on every run, so checks that must be identical belong in a script.

**What's worse from an AI test agent, a false pass or a false fail?**
A false pass hides a broken product, so it's the more dangerous one. A false fail has become more expensive too, because coding agents can now read CI results and try to fix whatever is red. A wrong failure can send one after a bug that doesn't exist.

**Can an AI test agent change the test it is grading?**
It shouldn't. Whatever grades a result should only report what it thinks should change. Any later edit to a test should be something your team configured, can switch off and can see in the test's history. A grader that rewrites the test it just failed can turn a real bug into a green.

**How do I check whether an AI agent's test verdict is right?**
Open the evidence behind it: the screenshot for each step, the agent's reasoning, the network and console logs, and the recording. Compare the path the agent took with the path the test describes, rerun the result if it looks wrong, and look at the test's pass/fail history.

**What does "unable to verify" mean in AI testing?**
It means the agent couldn't reach a confident pass or fail, for example because it couldn't log in, the environment wasn't ready, or the tests never reached the change. It should never count as a pass. Treat it as a task for a person.

**Should AI test agents replace Playwright or Selenium scripts?**
Not entirely. Scripts repeat the same steps on every run and cost nothing to license, which makes them right for the handful of flows you can't afford to break. An agent that understands the product is better suited to checking what each pull request changes, so running both covers more than either does alone.

**Who should make the release decision when an AI agent tests the build?**
A person. The agent's job is to gather evidence and give a clear verdict; the human reads it and decides.