Quality Assurance·

How to Replace Your Regression Suite: What to Replace, What to Optimize, What to Keep

A three-bucket plan for your regression suite: what to keep scripted on every change, what to optimize, and what to replace with testing per pull request.

QA.tech team

Your Regression Suite – a precision three-way mechanical sorting junction with three distinct glass channels.

You replace a regression suite by changing what it's for, and no reliable test has to be deleted along the way. Keep a small set of critical-path checks scripted and running on every change, optimize the slow, flaky and duplicated tests by rescheduling or merging them, and replace the long tail with dynamic testing of what each pull request changed.

What to replace in your regression suite, what to optimize, what to keep

Most arguments about a regression suite are about its size. We've made it in a large regression suite is a liability, and the definitions live in what is regression testing, so this piece skips both. It assumes you have a suite that mostly works, a nightly run that gets slower every quarter, and someone asking whether AI testing means you can retire the lot.

You can't. You shouldn't want to, either. If your regression tests already work, you've spent the time and money to build and maintain them, so keep them. A new toy arriving is no reason to throw out the things that work.

What changes is the job. Today your suite does two things at once. It proves that the handful of flows that pay the bills still work, and it tries to catch anything else that any change might have broken. It's good at the first job. The second makes it grow without limit, because every feature and every escaped bug adds a test and nothing is ever taken out.

Split those two jobs and each check you own lands in one of three buckets.

Type of checkBucketWhy
Sign-up, login, checkout, payment: flows that would stop a releaseKeep, on every changeA break costs revenue the same day. You want identical steps every time, blocking the merge.
Exact-value logic: prices, tax, ledger entries, permission rulesKeep, in unit and API testsThere's one right answer, and code is the precise place to check it.
Full passes that a regulator or a customer contract requiresKeep, on the release scheduleThe obligation doesn't go away. Run it where it's required.
Long end-to-end journeys that still catch real bugsOptimize: nightly or pre-releaseWorth having, too slow for the merge path.
Flaky testsOptimize: off the merge path, then diagnoseRandom red teaches people to ignore red.
Several tests covering one flow with small variationsOptimize: merge into one journeySame coverage for a fraction of the runtime.
Browser tests that assert a business rule by reading the screenOptimize: move the rule to a unit or API testWrong layer. The rule is cheap to check in code.
Detail checks on stable screens: labels, column order, a status colorReplaceThey only matter when someone changes that screen, which is when change-based testing looks.
Edge-case matrices: every locale, every input formatReplaceRun the matrix when the field changes. Nightly, it re-proves something nobody touched.
The new end-to-end test you'd add for each feature or bug fixReplaceTest it in depth on the pull request that introduces it. Promote it only if the flow joins the critical path.

"Replace" in that table never means delete. A replaced check stops being a permanent, scheduled script, and the flow it covered gets tested when a change touches it. The script stays in the repository, unscheduled, as a record of how the flow works. The only tests worth pruning from the suite outright are the ones for a feature that no longer exists.

What to keep: critical-path checks on every change

Start here, because this bucket is the answer to the coverage fear. It's a fixed floor, and it protects business-critical coverage while everything else moves. The keep set is the short list of flows where a break costs money or trust the same day. Can someone sign up? Can they log in and pay? Can they do the main thing your product exists for?

Tie each flow to a business number. If checkout breaks, revenue stops, and that's the test for whether a flow belongs on the list. A list that runs past one page has stopped being a critical path.

Whatever goes in has to be fast. DORA says developers should get feedback from automated tests "in less than ten minutes", and a blocking check that takes much longer gets bypassed. It has to be reliable too. The rule of thumb in Software Engineering at Google for what runs before a merge is "only fast, reliable ones", with everything else running afterwards.

And it should be boring, the same steps on the same data every run. This is what scripts are for. They're fast, they're deterministic and they cost nothing to license, so if yours are stable there's no case for rewriting them in anything newer.

Some of the keep bucket doesn't belong in a browser at all. Anything with one exact right answer is better tested in code, where it runs in milliseconds. If you're building a bank, unit test the ledger and the transactions, then let end-to-end tests cover the journeys around them.

What to optimize: slow, flaky and duplicated checks

Expect this to be the biggest bucket. It's also where the urge to delete does the most damage, because the trouble with most of these tests is where they run. They sit on the merge path, on every change, when they'd earn more on a schedule.

Slow. A long end-to-end run through a whole onboarding journey can still catch real bugs. Move it to nightly or pre-release, where the same Google chapter accepts "longer times and some instability, as long as you have proper mechanisms to deal with it." Once it's off the merge path, making the full run faster is work you can plan, because nobody's merge is waiting on it.

Flaky. Software Engineering at Google says tests "begin to lose value" as a suite approaches 1% flakiness. So a flaky test comes off the merge path today. Then someone works out which kind of flaky it is. The two kinds of flaky tests need different fixes, and a test that keeps tripping over a real race condition in the product is doing its job.

Duplicated. Suites get written in bursts. A team makes a push and writes a load of tests, stops looking after them, then makes another push a year later. You end up with six tests that log in, add to cart and stop at slightly different steps. Fold them into one journey with more assertions.

The other half of optimizing is running by risk, and you don't need a vendor's word that it works. Engineering at Meta reported in 2018 that predictive test selection caught "more than 99.9 percent of all regressions" while running "just a third of all tests that transitively depend on modified code". Meta trained a model on its own test history to get there. Test impact analysis is the version within reach of everyone else.

What to replace: the permanent test for every change

The third bucket is mainly tests you haven't written yet.

A suite that grows by one test per feature and one per escaped bug works as a minefield. You lay mines across the map and hope the next bug steps on one. It's only as good as the number of mines you've placed, and the bugs that reach production are the ones that walked between them. The scripted answer is to lay more mines. The other answer is to zoom in on the blast radius of each change: read the pull request, work out which user flows it can reach, and test those in depth on a running build before it merges. That's dynamic testing, and it's what takes over the long tail.

Take a change to the phone number field on your sign-up form. The first time, you want it tried with every country format, pasted with spaces, typed with a plus sign, abandoned halfway. That's the right depth for that pull request. You don't want the same matrix running every night forever. It gets slow and expensive, and the field hasn't changed since. Every repeat run is another line in the full AI cost of shipping a feature. Go deep once, where the change is, and again when something touches sign-up.

In practice an agent does the selecting, and how an AI agent knows what to test on a pull request comes down to what it's given to read. It starts with the test cases you already have for the flows the change touches, writes new ones only for a gap, and goes deeper the more the change can break. Some of the new test cases will deserve a permanent place, which is how PR test cases become your regression suite: promoted after the merge, a few at a time.

From the suite you already have, the detail checks move into this bucket. A label, a column order, the wording of an empty state. They matter when someone changes that screen and at no other time. Stop scheduling them.

Will you lose business-critical coverage?

This is the first thing engineering managers ask, and they're right to. Testing based on the diff has blind spots, and the plan only works if you cover them on purpose.

Start with the flow nobody touched. Change-based testing looks where the change is. It can't promise checkout still works when no pull request went near checkout but a config value, a third-party script or an expired certificate broke it anyway. The keep bucket exists for exactly this. It runs on every change and on a schedule, whatever the diff says, which is why regression testing in pull requests doesn't retire the scheduled run.

Then there's the legacy corner. If part of the system won't be touched for six months, no pull request will trigger a test of it in that time, and teams carrying decades of older code feel this most. Build a baseline up front. Have a person walk that area once, or point an exploratory agent at it, write the flows down as test cases, and put them in the scheduled run.

The third blind spot is the wrong guess. Whoever picks the tests for a change, a person or an agent, will sometimes leave out a flow that mattered. The guard is the keep set again, plus a stretch where the old suite and the new approach run side by side.

Tell the business one more thing. The big suite never covered everything either. Nobody can test every customer configuration on every deploy, because the run would cost too much and finish too late. So name what must never break, write it down where product and support can read it, and keep the happy path happy.

How to roll it out, starting this sprint

Steps one to five need no new tooling.

  1. Pull the run history. Take the last two or three months. For each test you want its runtime, how often it failed, and how many of those failures were real bugs. If nobody can say which failures were real, that's your first finding.
  2. Name the critical path with the business. Sit down with product and whoever owns revenue and ask one question: which flows would make us stop a release? Give each flow on the list an owner.
  3. Sort every test. One spreadsheet, one row per test: flow covered, runtime, failures, real bugs caught, duplicate of, bucket, owner. Put at least two people in the room. When you can't agree, choose optimize, because it's the reversible bucket.
  4. Make the keep set a blocking check. It runs on every pull request, inside DORA's ten-minute budget, and it's required to merge. If the keep set is flaky, fix that before you touch anything else.
  5. Reschedule the optimize set. Nightly or pre-release. Give each quarantined flaky test an owner and a date.
  6. Turn on change-based testing without making it a required check. It posts results on pull requests and blocks nothing. The old suite keeps running exactly as it was.
  7. Compare for a few weeks. For every bug found, record what caught it: the keep set, the scheduled run, the pull request test or a customer. That log is what you show anyone worried about coverage.
  8. Change the growth rule. A new permanent end-to-end test now needs a reason, and the reason is that the flow joined the critical path. Everything else gets tested on the pull request that changes it.

You'll know it's working when the merge check finishes in minutes, a red result on it stops the line without a debate, and the log shows no rise in bugs reaching customers. Give that last one a quarter.

If you have a QA team, the keep set and the calls in step three are theirs, and getting your QA team on board with AI testing starts with handing them that ownership. Without one, an engineer has to be named for the keep set, as in removing the dev-to-QA handoff.

Where an AI testing platform fits in this plan

Our platform can't import your existing scripts. Test case definitions export as JSON through the API or CLI, which gives you a description of each test case and no Playwright code, so the keep bucket stays where it is. Two runs of the same check aren't guaranteed to take identical steps, so anything that must execute the same way every time should stay a script. Web test cases run in a Chromium-based browser.

QA.tech, an AI testing platform, does the work of the third bucket. A pull request is the trigger. The agent reads what the pull request says and what it changes, along with the linked ticket, runs the existing test cases that exercise what changed, and adds new ones only for behavior nothing covers yet, all against the preview build. What comes back is pass, fail or unable to verify, and the pull request links to the run that keeps the screenshots, logs and trace.

The review picks its own test cases, and no setting forces a particular one into it. Critical-path enforcement therefore sits beside the review. Your scripted keep set can stay in its own CI job. If you have no reliable scripts for the first bucket, the platform can run that as well, as a small fixed plan of plain-language test cases, which the docs size at 5 to 10 critical test cases. That plan runs on every pull request, as a blocking Test Run Action on GitHub or through the Start Run API from a GitLab or Azure DevOps pipeline.

Requiring the review itself before merge is a GitHub-only option. Scheduled test plans, the nightly half of this plan, start on the Growth pricing tier. With no preview environment per pull request, the same review can run after merge against staging, at the price of the merge gate and clean attribution to a single change.

The primary difference between traditional script-based automation and a modern agentic QA workflow shows up in that third bucket. A script is written once and replayed. An agent's test case is chosen, or written, for the change in front of it, and the agent plans its steps from a goal each time, which is the line agentic testing vs traditional test automation draws. After a merge, a post-merge agent can promote the test cases worth keeping into the suite, labeled regression and auto-added, on GitHub and Azure DevOps but not on GitLab.

What all of this buys is confidence to merge. A developer, or a coding agent, hears what broke while the pull request is still open and the fix is cheap. The scheduled half of the plan is what automated regression testing with AI is for, and a team coming off Selenium can start with its flakiest tests, the route in migrating a Selenium suite to agentic testing.

FAQ

Can AI testing replace a regression suite?

Part of it. AI testing on pull requests takes over the long tail: the detail checks and the one-test-per-feature habit that make a suite grow. It doesn't take over the critical-path checks. Those keep running on every change, scripted if your scripts are already reliable.

What should stay in a regression suite?

The flows that would make you stop a release: sign-up, login, payment and the main thing customers use the product for. Add any full pass that a regulator or a customer contract requires. Logic with one exact right answer stays as well, in unit and API tests where it runs in milliseconds.

How do you reduce a regression suite without losing coverage?

Change when tests run before you change whether they exist. Move slow and flaky tests from the merge path to a nightly or pre-release schedule, merge duplicates, and keep a fixed critical-path set on every change. Then run change-based testing next to the old suite for a few weeks and log which layer catches each bug. That log tells you whether coverage dropped.

Should you delete flaky regression tests?

Take them off the merge path first, then find out why they flake. Some are badly written. Some share state with a test running in parallel. Some are failing on a real race condition in the product, and those are worth more than a green run. The only tests to remove outright are the ones covering features that no longer exist.

How does agentic testing differ from traditional automated testing?

A traditional automated test is a script: fixed steps, written in advance, replayed the same way on every run. An agentic test starts from a goal in plain language, and an agent works out the steps on the running application. Scripts execute faster and are fully deterministic. Agents cope with UI changes and can test flows nobody scripted. Plenty of teams run both.

How do autonomous testing agents find regressions?

They work out which user flows a change can reach, then exercise those flows on a running build of that change, so the regressions they find are the ones near the change. A regression in a flow the change never touched is outside their view, which is why a fixed critical-path set should run alongside.

Do regulated teams still need a full regression pass?

Yes. If a standard, an auditor or a contract requires a complete pass before release, nothing in this plan removes it. The plan changes what happens between releases: the mandated pass runs on the release schedule, a small critical-path set guards every merge, and change-based testing gives developers feedback while the pull request is still open.

Your team moves fast. Can your testing keep up?

Our QA agents test your product autonomously, so moving fast never means shipping broken. Start free or get a 30-minute demo.