Regression testing is re-testing features that already worked, to confirm a change didn't break them. You built something, or upgraded a dependency, or fixed a bug. Regression testing is how you find out what else moved.
It gets confused with retesting, which is narrower. Retesting means checking one specific bug you just fixed, to see whether the fix worked. Regression testing means checking everything around it that you didn't touch and don't expect to have broken. The second one is where the surprises live.
Most of this article is about a question nobody in this category wants to answer with a number: how big should your regression suite actually be?
Our answer, up front, because it's the whole argument. Keep a small scheduled suite of flows that must never break, and pair it with dynamic testing on every pull request, where the tests are derived from what actually changed rather than replayed from a script. Small suite, plus dynamic testing on the diff. Not one big suite running everything every time.
Here's the reasoning.
What regression testing covers
The triggers are mundane.
A code change ships, whether that's a feature, a refactor, or a one-line fix. A dependency or framework gets upgraded, which is the one that catches teams out, because nobody wrote the change and so nobody feels responsible for testing it. A config or environment change lands, new feature flag, new region, new payment provider. And a bug fix goes out, which needs both a retest of the bug and a regression pass around it.
The scope is always the same in principle: things that worked before, checked again. What varies is how much of "before" you check.
Regression testing vs retesting vs smoke testing
These three get used interchangeably in standups and they shouldn't be.
| What it checks | When | Typical size | |
|---|---|---|---|
| Retesting | One specific fix, on the exact steps that reproduced the bug | Right after a fix | One test |
| Smoke testing | That the build is alive at all: it loads, you can log in, the critical path works | Every deploy, often automated | A handful of tests |
| Regression testing | Features that already worked, to catch unintended breakage | After a change, or before a release | Anywhere from a dozen to thousands |
Smoke tests answer "is this worth testing further". Regression tests answer "did we break something we weren't looking at". If you want the distinction between smoke and sanity checks specifically, we've written that up in smoke vs sanity testing.
When to run regression tests
Teams settle into one of a few cadences, and they cost very different things.
On every pull request. The tightest feedback loop, and the one that catches problems while the person who caused them still remembers what they did. This only works if the run is scoped to the change, which is what dynamic testing does and what we come back to at the end.
Before every release. The most common pattern. Also the one that turns QA into a gate at the end of the process. One team we onboarded runs eight-week cycles, with regression and smoke tests squeezed into weeks seven and eight. Their QA lead's account of why: the developers keep coding at a speed the team can't keep up with, so somebody has to sit at the end and confirm it looks okay. That's a bottleneck with a person sitting in it.
Nightly or weekly on a schedule. Useful for catching environmental drift and slow-burn breakage. Also where big suites go to be quietly ignored, because nobody reads a report at 6am that's been amber for three weeks.
After deploy to staging. Underrated. The change is real, the environment is real, and you're testing what actually shipped rather than what you think shipped.
Most teams need two of these, not one. A tight scoped run on every change, and a fuller pass before release. The at-scale mechanics of the second one, parallelisation and build times, are covered in running full regression suites without slowing your builds.
Why regression suites get big, and what that costs
Nobody sits down and decides to have 1,300 regression tests. It happens the way debt happens, one reasonable decision at a time.
One fintech we talked to went straight for UI-based end-to-end tests early on, because that was the fastest way to get real certainty about whether the product worked. Right call at the time. Then, in their VP of engineering's words, maintaining it as they grew "just became bigger and bigger and bigger and more unwieldy." They ended up deprecating a lot of those end-to-end tests down into API and unit tests. The regression pack is still 1,300 tests, run every two weeks.
Three things go wrong at that size.
Flakiness stops being bad luck and becomes arithmetic. This is the part teams underestimate. Say each test has a 1% chance of failing for reasons that have nothing to do with your code, a timing issue, a slow response, a race condition. Run 100 tests and you'll see roughly one spurious failure per run. Run 1,300 and you're looking at a dozen. Every single one costs somebody time to triage, and none of them are bugs.
You can't engineer your way out of that with better tests, because the flakiness isn't in your tests. It's in the fact that you're driving a real browser against a real network. The internet is unreliable by definition. The only lever that actually moves the number is running fewer tests per change.
The usual response is to stabilise the suite by mocking more of it. That works, and it costs you the thing you were buying. Mock away enough of the world and the suite goes green without touching reality. That's confidence you haven't earned.
People stop believing the results. Once a suite has a persistent handful of red tests, the team learns to read past them. We've watched this happen from the vendor side, which is an uncomfortable seat: the tests are running, the failures are real failures, and nobody's looking. At that point it isn't a safety net any more, it's just a thing that runs because it always has.
The maintenance bill comes due, and nobody pays it. Ask who owns updating the tests when the UI changes and you usually get a pause. We put that question to a QA lead at a medical-device company. When someone changes a feature and it breaks the tests, "we have no time to fix those." That isn't negligence. It's what happens when suite size grows faster than the team, which it always does. We've written about the shape of that cost in the hidden tax of test maintenance.
There's a fourth thing, and it's the one that surprised us most. A fair number of "automated regression suites" aren't automated. One team we onboarded runs their regression through a test-management tool wired into Jira. Nine times out of ten, their engineering lead told us, the test case says do A, do B, do C, and then a person opens a browser, clicks through it, and marks it done. Strictly speaking they have no evidence anything was tested at all. Just someone's word that it was.
If that sounds like your setup, the suite size isn't your problem. The suite is fiction.
How big should a regression suite be?
Small enough that a red test is worth investigating.
That's the whole criterion. If a failure makes someone stop and look, the suite is the right size. If a failure makes someone shrug, it isn't, and adding more tests will make it worse rather than better.
In practice that means the regression suite holds flows that must never break, and nothing else. Can someone sign up. Can they pay. Can they do the main thing your product exists to do. For us internally that's around 60 tests.
The suite is only half the answer, though, and this is the part that makes a small number safe rather than reckless. The other half is dynamic testing on every pull request: tests derived from the change in front of you, running against a preview environment, instead of a fixed set replayed on a schedule. One of our customers runs exactly that shape, a core regression suite in the tens of tests plus roughly 20 dynamic tests per change. Sixty tests on their own would be thin coverage. Sixty plus dynamic testing on the diff covers more than 1,300 scheduled tests do, because the 1,300 were written against a product that has since moved.
Two rules we apply to our own suite, and both of them are about what stays out.
Nothing goes in because it might be useful. Our founder's version of this, on an internal call, was that he doesn't want a suite checking whether the status dots are green in a particular view. It's too granular, it costs more than the information is worth, and he doesn't want a slightly-off shade of green blocking a release. The bar is "must never break", not "would be nice to know".
And nothing goes in just because it was written. Tests accumulate from feature work, and the default of promoting all of them into the regression pass is how you get to 1,300. Most tests are worth running once, against the change they were written for, and then rarely again.
If you want the harder version of this argument, when to kill a test goes further, and why 100% test coverage isn't the goal covers the measurement side.
What to do with the tests you take out
Don't delete them. This is where the argument needs a counterweight, because "keep the suite small" can slide into "have fewer tests", and those aren't the same claim.
A test that runs rarely still does two useful things. It documents a flow, in a form that's more precise than a wiki page and more current than a spec. And if your testing is agent-driven, it teaches the system what your product does. One of our own team put it as a chicken-and-egg problem when a customer asked which to build first: pull-request testing gets better the more tests exist, because every test adds to the model of how the app behaves, even the ones nobody runs on a schedule.
So the shape is a large library, a small suite, and dynamic tests on top of both. Most tests exist, few tests run every time, and the ones that run against a given change are chosen by that change. The distinction people miss is that "in the suite" and "exists" are different states, and only one of them bills you every release, which is the part that matters when you're deciding what to cut.
The mechanics of keeping that library maintainable are in best practices for automating regression tests.
The other half: dynamic testing on the diff
Everything above is true regardless of what tool you use. Here's what changes when the tests aren't scripts.
First, a definition, because the term gets used two ways. Classically, dynamic testing just means running the software rather than reading it, which makes every regression test a dynamic test and the word not very useful. The sense that matters here is narrower: the test itself is dynamic. It's generated against the change in front of it rather than written once and replayed. We've unpacked both meanings in what is dynamic testing.
Two of the three costs of a big suite come from selectors. A scripted test points at specific elements, so moving a button breaks it, and someone has to go fix the pointer. That's the maintenance bill, and it's also a good share of the flakiness.
The category's answer to this is self-healing: detect that the selector broke, guess which element you meant, carry on. Look at what that feature admits. Something had to break for there to be anything to heal. The tests are coupled to your implementation, they'll break every time you ship, and the vendor has built machinery to make that cheaper to live with rather than removing the coupling. And because the re-guess is probabilistic, the failure mode is the bad one: when it guesses wrong, the test goes green while checking the wrong thing. A feature whose whole purpose is keeping tests green after the application changed is, architecturally, a machine for suppressing your only alarm. We've made that argument at length in the top 5 things everyone gets wrong about self-healing tests.
Removing the selector is the other option. A QA.tech test is a goal written in plain language: "a new user can sign up, add a card, and get charged the right amount." There's no code and no element to point at. On every run the agent reads the screen and works out how to reach the goal in the app as it currently is. Move the login into a modal and nothing needs healing, because nothing was ever mapped. What does still need a person is a change to the flow itself: if login starts requiring a second factor the test never knew about, someone updates the test. That's the honest limit. It's also a better trade, because it triggers when the product actually changed rather than every time a class name did.
The bigger change is selection. If you can generate tests against a specific change cheaply, you don't need a big suite to be thorough. Dynamic testing on every pull request means classifying what changed, generating tests for it, running them against the preview environment, and posting a verdict before a human opens the review. Because the agent reads the rendered screen rather than a selector map, those tests can be written for a change nobody has tested before, which is the thing a scripted suite structurally cannot do. The verdict is anchored to the diff, so pre-existing failures don't block your PR, and if the tests ran without actually exercising your change you get told that rather than a false green.
Then the small core suite runs before release, as a pre-release check that compiles the regression pass and an exploratory review of what changed into one report.
So the recommendation, stated plainly: a small scheduled regression suite for the flows that must never break, plus dynamic testing on every pull request for everything else. Not fewer tests overall. Fewer tests per run, chosen by what moved.
One more thing worth saying, because of the traceability problem earlier. Every run produces a step-by-step trace: what the agent reasoned, what it clicked, screenshots, video, network and console logs. If your current evidence that regression testing happened is a person ticking a box in Jira, that's a meaningful upgrade regardless of the tooling debate.
If you want the mental model behind goal-based tests rather than the product pitch, designing tests for intent, not selectors is the longer version. And one customer's numbers, for scale: they replaced 320 hours of manual testing a month.
Where to start if your suite is already too big
Don't rebuild it. Two moves. Pick the flows that would make you stop a release if they broke, and put only those in a scheduled suite. Then wire dynamic testing into your pull requests so everything you took out is still covered, against the change that touches it, rather than on a timer nobody reads.
Then watch one number for a month: how often does a red test turn out to be a real bug? If that's above half, your suite is roughly the right size. If it's under a fifth, you're paying people to triage noise, and no amount of additional coverage fixes that.
The harder question is what you're not testing at all, which a suite can't tell you by definition. That needs a map of the application rather than a list of tests: which pages and flows have been exercised recently, which have never been touched, and which gaps are worth closing first. We build that into the product as a coverage view, currently in beta. Whatever tool you use, the number worth tracking is not how many tests you have. It's how much of the product they actually reach.