Contents
A flaky test is one that passes and fails without the code under it changing. Run it now, it's green. Run it again, red. Nothing shipped in between. Everyone who has owned a test suite knows the feeling, and the standard advice is the same everywhere: hunt flakiness down and eliminate it, because a suite you can't trust is a suite people start ignoring.
That advice is half right, and the missing half is where teams quietly lose coverage. Because "the result changed and the code didn't" actually describes two different things, and only one of them is a defect. Treat them as the same problem and the fix for the first one blinds you to the second.
Why one flaky test in a hundred is a suite-wide problem
Start with why flakiness gets treated as an emergency, because the reaction is rational even when the conclusion isn't.
Take a suite of 100 tests, each of which fails 1% of the time for no real reason. That sounds tolerable per test. Run the whole suite and a fully green run happens only about a third of the time (0.99 to the power of 100, roughly 0.37). So closer to two runs in three, something is red and nothing is wrong. That's arithmetic rather than a measured figure, but the shape is what every team with a big suite lives. Developers learn within a week that red usually means nothing, they start re-running until green, and the suite stops being a signal. The test that was protecting checkout is now indistinguishable from the forty that cry wolf.
There is a second thing hiding in that arithmetic, and most teams walk straight past it. The reason all hundred ran is that nobody knew which five mattered. You changed something that touches five tests, you ran a hundred, and you bought ninety-five tests' worth of flake risk without buying a single piece of extra information about your change. Seen that way, the cheapest lever on flakiness has nothing to do with hardening individual tests. It is scope: not running the ones your change couldn't have touched.
So the instinct to drive flakiness to zero is correct. The mistake is in what you reach for to do it.
Kind one: noise. Kill it.
The first kind of flake is the test result moving when neither the product nor the behaviour you're checking moved. A fixed wait(2000) that's long enough on your laptop and too short in CI. A test that passes alone and fails in parallel because it and its neighbour both write to the same account. A selector that matches two elements and picks whichever rendered first this time. An animation that hadn't finished. A network call that hadn't come back.
None of that is the product telling you anything. It's the test binding itself to details that were never part of what you meant to check: exact timing, render order, the incidental structure of the markup, the state left behind by whatever ran before. This kind of flake is a bug in the test, and it deserves the standard treatment. Wait for conditions, not clocks. Isolate state so tests don't collide. Assert on what the user sees, not on the twelfth div. Quarantine the ones you can't fix yet so they stop poisoning the signal, and actually go back and fix them.
Most of this kind of flake is brittleness, and brittleness comes from the test being pinned to things a user never cares about. A person doesn't fail to log in because the button moved four pixels or because the DOM node got a new wrapper. A script does, because the script was a binding to the old structure. Which is why the tool you test with changes how much of this you even have.
An agent that works from a goal rather than a script removes most of kind-one flake at the source, because it was never bound to the brittle details in the first place. It waits for the page to settle before it looks, the way a person waits for a spinner to stop. When something takes longer than usual it's smart enough to just wait a bit longer, or reload and go again, rather than asserting into a half-rendered page. It finds the submit button by what the button is and where it sits on the screen, not by a selector that breaks when a class name changes. The timing races and selector breakage that make up the bulk of a Playwright suite's flake list mostly don't have anywhere to attach. You didn't write the waits, so you can't get them wrong.
None of this makes agents magically stable. There is always some flake risk in a web application, because of how the web works, and if you run a big enough sweep you will always find something misbehaving somewhere. The point is narrower: the specific mechanical causes of most scripted flakiness are things a reasoning agent handles the way a manual tester handled them, without you writing and maintaining the handling.
Kind two: signal. Don't suppress it.
The second kind of flake looks identical from the outside. Green, then red, and the code didn't change. But this time the run went a different way, and the different way found something real.
This is where a lot of teams reflexively reach for the wrong fix, because the reflex from kind one is now a habit. An agentic test doesn't march down one frozen path. Ask it to check that a user can create a record, and it creates a real one, every run. Run that a dozen times and it hits a list that's now long enough to paginate, and pagination was never tested, and it's broken. Run it a thousand times against an account that keeps growing and something gets slow. A real user who does this daily would have hit exactly that. Your deterministic script, starting from the same clean empty state every single time, would never have.
None of those are the test being unreliable. They're the test doing the job you can't afford to do by hand: walking slightly different, valid paths and finding the long tail, the bugs that only show up for the customer who's been on the product for eight months and never files a ticket. Freeze the path to make the test repeatable and you delete the only mechanism that was going to catch them.
This is the distinction the word "flaky" hides. Kind one is the result changing when nothing real did. Kind two is the result changing because the run touched something real that your happy path never does. Same symptom, opposite meaning. A team that has trained itself to treat every non-deterministic red as noise will stamp out both, without noticing the second one was the valuable kind.
Telling them apart when a red lands
The whole thing rests on a classification you have to make in the moment: a run went red, which kind is it? The answer is in the trace, not in the colour. Re-run it as it was. Kind-one noise usually won't come back, because the timeout or the stale element or the parallel collision that caused it was incidental. Kind two reproduces the moment you recreate the state it found, the long list or the grown account, and the trace shows the agent walking a valid path into a real defect: an actual error, a wrong result, a slow response, not a mechanical stumble.
Most of that shouldn't be your job. A tool worth using does the re-running itself when a test fails, several times, and works out whether the test is being odd or whether there's an actual bug, then tells you which. What you're left to judge is the small pile it couldn't call. That's also why an auditable per-step trace matters more here than it ever did for a script: screenshots, the reasoning that picked each action, the network and console output at that moment. A red no engineer can explain from the trace is a tooling problem first.
The trap: determinism stops being a means and becomes the goal
Kind-one flake is real and painful, so the team declares war on non-determinism. To win it, they pin everything: fixed data, frozen clocks, one hard-coded path per test, no variation anywhere. The suite goes reliably green. Trust comes back. And nobody notices that the suite now only ever checks the exact narrow lane it was pinned to, because the thing that used to wander off that lane and find problems was the same non-determinism they just spent a quarter eliminating.
That's determinism turning from a means into the goal. A perfectly deterministic suite gives you what one of our own founders calls, on customer calls, "some sense of security" while it quietly stops "testing the new things." It's guardrails against catastrophic breakage, which is worth having, and it is not the same as knowing your product works. The most dangerous outcome of over-optimising for determinism isn't a slower suite. It's a suite that is always green and increasingly blind.
False positives and false negatives, since that's what people actually ask
When someone hears "non-deterministic testing," the fear splits into two, and they're worth separating because the answers are different.
A false positive is the test raising the alarm when nothing is wrong. This is the one people fixate on, because it's loud and annoying, and they describe it very concretely. A QA lead at a consumer app put the worry to us as: does it just come back with unrelated timeout errors, telling me I can no longer sign up for an account because account creation took five and a half seconds instead of five?
Most false positives are kind-one noise, and the fix is the fix above: at the source for agents, with hygiene for scripts. The genuinely new source of false positives with an agent is that the thing judging the outcome is itself a model, so it can occasionally call something broken that a person would wave through. That isn't a third kind of flake, it's another source of false positives, and it gets handled like noise: bounded, never trusted on a single run.
You bound it three ways. The first is a reproducibility policy, so a single red never gates a merge on its own. Re-run it. Block the merge only on repeated fails for the same goal. Quarantine any goal that flakes more than the Playwright suite you already live with. The second is a pinned check for where a judgement call isn't good enough: put an explicit assertion inside the goal, so a specific request has to return a specific status and body and there's nothing for the model to be fuzzy about. The third is measurement. Track the real-red rate and publish it, because a number the developers can see is what earns the verdicts their attention.
A false negative is the quiet one: the test says fine, and it isn't. This is the dangerous direction, and it is the one that chasing determinism actively causes. A frozen script that always takes the same path is, by construction, silent about everything off that path. Worse is the self-healing script that hits a changed UI, re-guesses which element you meant, guesses plausibly, and reports green while checking nothing you'd recognise. That's a false-negative generator dressed up as resilience, and it's why the heal is the problem, not the fix. Variation is the antidote to false negatives. A test that occasionally walks a different valid path is a test that occasionally checks something your frozen one never will.
Put the two together. Kind-one noise inflates false positives, so you kill it. Kind-two variation is what protects you from false negatives, so you keep it. Confuse them, kill both, and you've traded a loud, honest problem for a quiet, invisible one.
"But how do I know it runs the same test next time?"
This is the fair version of the objection, and engineers put it to us almost word for word. With code-based tests and locators, one VP of engineering told us, it's absolutely deterministic, it will follow the same path every time; but if the test is steps an agent interprets, you can be strict with it and still get flakiness. If an agent decides for itself which flows a change affects, two runs on similar changes might not exercise exactly the same set. And the worry underneath is real: you found an edge case three pull requests ago, you wrote it down precisely so it would never slip again, and non-determinism sounds like it reneges on that promise.
Two things. First, the variation is bounded, not wild. An agent is a statistical model, not a random one. Ask it to go through a sign-up flow and it goes through the sign-up flow, every time; what varies is the incidental how, not the what. It won't wander off and start testing your billing page because it felt like it. What keeps it inside those walls isn't a promise, it's the setup: a goal it has to satisfy, a step budget that stops it before it drifts, and a run scoped to the part of the app the change touched.
Second, anything that genuinely must run identically every time can be made to. Pin it. Keep the checks that have to be byte-identical, the ones asserting an exact number or an exact contract, as deterministic tests in code, and keep the edge case you fought for as a named goal that runs on every change to that area. You are not choosing between "all deterministic" and "all agent decides." You're deciding, per check, which behaviour you want.
And the concession that matters: an agentic suite is not going to be 100% deterministic, and if that's a hard requirement for a particular check, that check should be a script. The argument here isn't that determinism is bad. It's that determinism is a tool you apply where it earns its place, not a virtue you maximise everywhere.
What to actually do
The shape that works isn't one or the other. It's two layers doing different jobs.
Keep a small, deterministic regression suite of the flows that must never break, run on a schedule, written to stay green and mean it. Unit tests, contract tests, snapshot tests, the handful of critical paths. Byte-identical, fast, in code. A tester will tell you that when you run a regression suite you want it doing exactly the same thing over and over, so you have something stable to compare against, and they're right. Regression is where sameness earns its keep. Keep this layer tight, on the things that actually matter, and leave it exactly where it is after you adopt anything agentic.
Then run dynamic, goal-driven testing on every pull request, scoped to what the change touches, and let it vary. This is the layer that walks the long tail, that hits pagination on run ten and the slow query on run a thousand, that checks the flows you were never going to script because scripting them by hand was never worth it. The point isn't identical runs. It's catching what the guardrail misses.
Scope is doing double duty here, and it's the part teams underuse. Running only what your change could plausibly have broken is what makes the second layer affordable, and it's also the single biggest reduction in flake you can make without touching a test, for the reason the arithmetic showed at the top: every irrelevant test you run is flake risk you took on for no information.
Variation has a cost, and the cost is triage. Some reds will be a path you don't care about, and someone has to glance at them to know that. Scoping the run to the diff is what keeps the bill down, because the wandering happens on the surface you just changed rather than across the whole product. Measure it the way you measure the noise: the share of investigated reds that turned out real. If that share holds, the triage is what's buying you the long tail, and it's cheaper than the incident it prevents.
Kill kind-one flake wherever it lives, in either layer. Protect kind-two variation in the second layer as the feature it is. And whichever tools you use, publish your flake numbers to the developers you're asking to trust the results, because a number they can see is the only thing that earns that trust.
Where QA.tech sits in this
QA.tech is an agentic testing platform: tests are goals in plain language, and the agent drives a real browser or mobile emulator, waits for the page to settle before it acts, retries the way a manual tester would, and judges the outcome on every run. That's what takes most of kind-one flake off the table without you writing waits or selectors. Re-runs on failure are configurable, so a test that goes red can be tried again and the flaky reds separated from the real ones before the result reaches you. Runs are bounded, a step budget stops a run rather than letting it wander, and where a verdict needs to be exact you pin a network check inside the goal so there's nothing for the model to be fuzzy about. It runs on every pull request against your preview environment, scoped to the diff, and posts a verdict before merge, which is the dynamic testing layer above. The deterministic guardrail, the flows that must never break, is the regression suite you keep alongside it. If you want the mechanism underneath, how an agent perceives, plans, acts and verifies, that's in what is agentic testing.