Quality Assurance·

How PR Test Cases Become Your Regression Suite

What happens to PR test cases after the merge: which ones earn a place in the regression suite, how stale ones get updated, and how to keep the suite small.

QA.tech team

From PR Tests to Regression – a selective brass ratchet feeding one glass cartridge into a compact ordered rack.

PR test cases become a regression suite one merge at a time, and only a few of them should make it. After a pull request merges, the ones that cover a main user flow, passed, and don't lean on that PR's data are promoted, and existing test cases the merge made wrong are updated or parked. Everything else stays out, and plenty of merges promote nothing.

How do PR test cases become a regression suite?

The tests from a pull request and the tests in a regression suite have different jobs. A PR test case answers one question once: does this change work? A regression test case answers another on every run, for as long as you keep it: does this flow still work? Moving a test case from the first job to the second is a promotion, and it should be a decision each time.

Promotion starts once the review is over. Picking and writing test cases before the merge is the other half, how an AI agent knows what to test on a pull request. Here the PR is merged, and somebody has to decide which of its test cases your team will still be running in six months.

The path has four steps.

  1. The PR opens and gets its test cases. Some exist already and are selected because they touch the change. Others are written for behaviour nothing covers yet. They run against the PR's build before the merge, because a result that arrives after the merge can't stop it.
  2. The new ones stay provisional. They belong to the PR. They aren't in the suite and can't turn tomorrow's regression run red.
  3. The merge triggers a sort. Each new test case is promoted or left as a draft. Each existing one the merge touched is kept, updated or parked.
  4. The suite changes a little, or not at all. Whatever was promoted is marked, so you can see later what was added this way.

Steps 1 and 2 are regression testing in pull requests. Step 3 is where suites go wrong, in both directions. Promote everything and the suite swells with test cases nobody has read. Promote nothing, ever, and each PR pays again for coverage an earlier one already worked out. Engineering leads ask us for the middle path, a way to promote some PR test cases into the main suite so nobody generates piles of them that never run again.

Promotion only settles which new test cases join. What the suite should hold in the first place, a short critical-path set on every change with everything else tested per pull request, is the plan in what to replace, what to optimize and what to keep in your regression suite, and what regression testing is has the background.

The promotion rule: when a PR test case earns a place in the suite

A promotion rule is a short list of questions a test case has to pass before it joins the suite. Write it down once and apply it at every merge, whoever or whatever does the applying.

QuestionPromote whenLeave as a draft when
Does it cover a flow a user would notice breaking?It walks a main flow end to end: sign-up, checkout, the core path of a new featureIt checks an edge case, an internal admin screen, or a detail that only mattered for this change
Did it pass on the merged change?Its last run against the PR's build passedIt was last seen failing. That's a bug to fix or a bad test case
Is it free of PR-specific data?It creates or finds its own data on every runIt depends on a record, ID, name or URL that existed only for this PR
Is the flow already covered?Nothing in the suite checks the same behaviourAn equivalent test case exists. Update that one if it needs it
Will it run on its own?Its prerequisites, such as a login, are in the suite tooIt only works in the order the review happened to run things
Would it survive a refactor?It checks what the user sees and what the requirement saysIt checks markup, copy or layout that can change while the behaviour stays the same
Does someone own it?A team is named for the day it failsNobody would know whose problem a red run is

The refactor row borrows its bar from Software Engineering at Google, which says "the ideal test is unchanging": once written, it shouldn't need to change unless the requirements do. A PR test case that would break on the next CSS cleanup doesn't meet it.

The duplicate row matters more when agents write the test cases. Evan X. Merz, a developer writing about agent-heavy codebases, describes the pattern: "When AI writes e2e tests, they tend to only cover the cases which are already covered in the codebase." So check for an equivalent before you promote, or the suite fills with near-copies.

A test case an agent promoted overnight has no author to ask when it goes red. That's what the last row is for. In its chapter on larger tests, the same book asks who maintains a test and who diagnoses it when it breaks: "Without clear ownership, a test rots." Name the team at promotion time.

What each kind of merge should do to the suite

Software Engineering at Google sorts changes to production code by kind and says how tests should react to each. It was written about unit tests. The logic carries over to end-to-end test cases at the merge.

Merged changeNew test casesExisting test cases
Refactor with no behaviour changePromote noneLeave them alone. If one broke, it was tied to the implementation or the refactor changed behaviour after all
New featurePromote the main-flow test case and skip the edge casesLeave them alone
Bug fixPromote the test case that reproduces the bug, if the flow mattersLeave them alone
Deliberate behaviour changePromote only if the new behaviour is a main flowUpdate the ones that describe the old behaviour. Park the ones for anything removed

Refactors, dependency bumps, copy tweaks, fixes for a bug an existing test case already catches: none of these should add a test case. The bug-fix row is the one to protect. The book's position is that "the bug fix should include that missing test case", and a test case that reproduces a real defect in a flow you care about has already shown it can catch something.

Only the last row changes test cases you already have. That's the harder half of the job.

What happens to test cases that go stale after a merge?

A test case goes stale when a merge changes the product on purpose and the test case still describes the old behaviour. Checkout drops a step. The settings page moves under a new menu. The next regression run fails, the product is fine, and someone loses a morning working that out.

Update the test case when the flow still exists and now behaves differently by design. Park it, meaning move it out of the suite into a draft or quarantine state, when the feature it covers was removed. Deleting is tempting and it's the wrong default, since a parked test case can come back when the removal turns out to be half-finished.

The merge is the right moment for both. It's the one point where you know the behaviour changed deliberately and you're holding the diff that shows why.

The risk sits in the update. An update whose only purpose is to turn a red test green is worse than none, and it's an easy one for an agent to make, because loosening a check is a shorter route to green than understanding the change. Daniel Mauno Pettersson, our co-founder and CEO, said as much to a team running a home-built loop that writes its own tests: "you have to trust the process of your regression test being updated in the right way and actually testing the changes in the right way."

That trust rests on conditions you can check:

  • The update is tied to the merged diff. If nothing in the diff explains the new behaviour, the test case is failing for another reason and shouldn't be edited.
  • Whatever grades a run doesn't edit the test case it's grading, which is part of the case in whether you can trust an AI agent to run your tests.
  • Every edit leaves the previous version behind for comparison.
  • Parking takes stronger evidence than updating, and nothing is deleted without a person.

None of this is the same as a tool that re-guesses a locator in the middle of a run so that a failing step passes. That happens while the test is being graded, and a wrong guess comes back green. An update at the merge changes what the test case says, once, with the diff as its reason.

Which platforms auto-update tests from a pull request?

It depends on where the tests live. If they're code in the repository, say a Playwright spec next to the feature, whoever changes the behaviour updates the spec in the same pull request. That setup runs quickly, gives the same answer every time and costs nothing in licences, and if it's reliable for you there's no reason to replace it. Tools that repair locators during a run don't read the pull request at all. The third kind is a testing platform that runs a step after the merge, which reads the merged diff, updates the test cases the merge changed and promotes the new ones worth keeping.

The product we build is the third kind, and it has limits. There's no post-merge agent on GitLab, where a review run after the merge tests the merged code and doesn't promote anything. Where the agent exists, it never archives or deletes a test case by itself. A promoted test case carries a label and, today, no structured link back to the pull request it came from. And a PR that was closed without merging, or never had a completed review, gets no post-merge step.

QA.tech, an AI testing platform, runs that step as a post-merge agent on GitHub and Azure DevOps. It works in the same conversation as the PR testing review, so it starts with the test cases that ran, their results and the diff. The review is the main event, because it's what lets a team merge with confidence. This step is the upkeep that follows it. If you're shortlisting an AI testing tool with GitHub PR integration, ask each vendor what happens at this point. The PR comment is the part every demo shows.

Post-merge agent defaults on GitHub, Azure DevOps and GitLab

PlatformPost-merge agentWhat you do
GitHub, new integrationOn, with the default prompt prefilledNothing, unless you want to edit the prompt
GitHub, existing integrationOff until you enable itTurn on Run post-merge agent under Settings → Integrations → GitHub App
Azure DevOpsSupported, off until you enable it on the repositoryEnable it and add the "Pull request merge attempted" service hook
GitLabNot availableActivate and update test cases yourself

The prompt decides what it does

The agent works from a prompt you can edit, and it changes test cases only when the prompt asks. Before it promotes or creates anything, it checks for an equivalent test case. The default prompt in QA.tech reads:

Now that this pull request is merged, update tests the merge changed, promote the main-flow candidate tests that passed during review, and convert clearly stale tests to draft. Skip edge cases. Never keep tests with hard-coded PR-specific data.

Edit the prompt and the behaviour follows. You can restrict it to flows a normal end user would hit, have it add a promoted test case to the Smoke test plan when the flow is critical and the test case is stable, or tell it to leave test cases alone and open a tracker issue. Promotion itself only activates and labels. The agent adds a test case to a test plan when the prompt asks for it and gives the criteria, and scheduled test plans are a Growth-plan feature.

A merged PR, step by step

Here's an illustration. A PR adds saved filters to a search page and retires the old advanced-search screen. During the review, new test cases are written as drafts and kept out of your test list. After the merge, with the default prompt:

  • A test case that saves a filter and applies it again is a promotion candidate. This PR's review created it, it's still a draft, and it wasn't last seen failing. Promotion activates it, along with the login test case it depends on, and labels it regression and auto-added.
  • A test case for a filter name at the length limit is an edge case. It stays a draft.
  • One that searches for a record the author made by hand carries PR-specific data, so it isn't promoted.
  • An existing test case for search results still sets its filters the old way, through a step the merge removed. It's updated to match the merged state.
  • The test cases for the retired screen are converted to draft, only on strong evidence in the diff, and together with any chain of test cases that depends on them.

Had the same PR been a refactor, that list would be empty. The docs' wording is "Promoting nothing is a normal outcome."

A draft is skipped whenever a test plan runs, so nothing left behind can fail your suite, and a later review can still pick up a relevant draft when another PR touches the same flow. To see what was added for you, filter the test list by auto-added. Every edit to a test case is kept as a revision you can compare.

How do I keep test coverage up to date when agents ship code daily?

Promote selectively at every merge, and prune on a schedule. Coverage stays current because each merge adjusts the suite by the small amount it deserves. The suite stays runnable because something also leaves.

The failure mode is easy to reach. We've seen a developer wire up an agent of their own so that every closed PR produced test cases. It ran. Nobody had time to read what it wrote, and before long the same team's test lead was looking at a suite that had gone from too thin to overkill, much of it created from merged PRs, with no way yet to verify that each test case did what its title said. A test leader at a much larger company told us where that ends: let the AI write test cases for everything and eventually you can't run them all.

Generating test cases after a merge is the easy half. Left alone, a generator is happy with what it did. Somebody still has to read the result, and a test case existing no longer proves anyone thought it was worth writing, which is the shift behind the testing pyramid in the age of agentic AI.

A few habits keep the count in check, and none needs a particular tool.

  • Label every test case that arrived automatically, so "what did the agents add this month?" is one filter away.
  • Run a short weekly pass over newly promoted test cases with the team that owns the flow, ending in keep, fix or park for each.
  • Give the suite a run-time budget. A promotion that would break it has to displace something.
  • Park a test case that fails when nobody can say why. Don't count on authors to do it by hand. Slack reports that automating the detection and suppression of flaky tests took test job failures from 57% to under 5%.
  • Prune, using the criteria in when to kill a test. DORA's list of common pitfalls in test automation includes "Failing to curate your test suites."

Coverage as a percentage won't tell you whether this is working, for the reasons in why 100% test coverage isn't the goal. A plainer number does the job here: how many test cases joined the suite last week, and can someone explain each one?

FAQ

Should every PR test case be added to the regression suite?

No. A PR test case checks one change once, and that's a lower bar than checking a flow for as long as the product exists. Promote the ones that cover a main user flow, passed on the merged change, carry no PR-specific data and don't duplicate something already in the suite. Leave the others as drafts. A merge that promotes nothing is normal, especially a refactor or a fix for a bug an existing test case already catches.

What is a post-merge agent?

A post-merge agent is an automated step that runs after a pull request merges and maintains the test suite using what the PR's review produced. Its usual jobs are promoting new test cases worth keeping, updating existing ones whose behaviour the merge changed, and moving ones the merge made obsolete out of the suite. It should work from instructions you can read and edit, and it should leave a record of what it changed.

Who owns a test case an agent added to the regression suite?

The team that owns the flow it covers. A test case that was promoted automatically has no author to ask when it fails, so name the owning team at promotion time and label the test case as automatically added. If no team will take it, it doesn't belong in the suite.

What happens to PR test cases that aren't promoted?

In the setup described here, they stay as drafts. A draft isn't part of the suite and doesn't run on a schedule, so it can't fail a regression run. It still has value. When a later pull request touches the same flow, the draft can be picked up and run again, and a person can activate it at any point. Drafts do need an occasional clear-out, because a growing pile of them is its own kind of clutter.

How often should promoted test cases be reviewed?

Weekly is a sensible default for a team that merges every day. Filter for what was added automatically since the last pass. For each one, check that it asserts something a user would care about, that it has passed on its recent runs, and that a team owns it. Anything that fails those checks goes back to draft. The pass is short when promotion is selective. If it starts taking an hour, the promotion rule is too loose.

Your team moves fast. Can your testing keep up?

Our QA agents test your product autonomously, so moving fast never means shipping broken. Start free or get a 30-minute demo.