Regression testing in the pull request: catching regressions before merge

Regression testing re-checks that the things that worked still work after a change. The usual place to run it is after the change has merged: a nightly plan, a Friday run, a pass before the release. The pull request is earlier than any of those. It's also the only moment when the regression is still one change wide.

Why the nightly run stopped being the safety net

The nightly suite was designed for a world where integration was slow. Martin Fowler's 2024 rewrite of Continuous Integration still describes the shape: a commit build that "has to be done quickly", and a secondary build that "may not run after every commit" because "this suite might take a couple of hours to run". That was fine when a release carried a dozen changes.

It stopped being fine when the number of changes went up tenfold and the number of people checking them didn't. An engineering lead on a ten-year-old web platform told us in September: "it used to be 15 PRs per release, and now it's 150, 200 PRs per release." His phrase for what that does to production was "a sketchier dice every time."

Faros AI's 2025 report on more than 10,000 developers found that teams with high AI adoption merge 98% more pull requests, and that PR review time rose 91%, which the authors call "a critical bottleneck: human approval." CircleCI's 2026 State of Software Delivery, on 28.7 million workflows, puts main-branch success at 70.8%, the lowest in over five years, with 72 minutes for a typical team to get back to green. Feature-branch throughput went up 15%; main-branch throughput went down 7%.

A nightly suite that runs against main after forty of those PRs have landed has one answer when it goes red: something in the last day broke something. "If it breaks something, right now we catch it when it gets merged to staging." That was a co-founder at a meeting-software company, on a suite that only runs on staging. By then the author has moved on. The fix starts with working out which PR it was.

What the nightly suite actually tells you

Two things, and neither is what people want from it. The first is that main is broken somewhere, with nothing pointing at the change that did it. And a suite that runs everything reports every environmental wobble along with the real breaks. A QA lead running daily and weekly suites: "part of the time of my team is getting drawn by just checking the flaky failures." Google's testing team put numbers on this across 4.2 million tests in 2017: 14% of their large tests were flaky, against 0.5% of small ones, and in their 2016 analysis, 84% of the transitions they saw from pass to fail involved a flaky test. The large, browser-driven tests a regression suite is made of are the ones that flake.

The second is that it only ever checks what someone wrote a test for. Vilhelm von Ehrenheim, our co-founder and Chief AI Officer, said this on a webinar in September, on why a change to something like a payment flow needs more than the suite can give: "the normal classic regression suite that runs after a merge or something, is typically very poor at handling some of these things. There might be much more nuanced, smaller detail, and you need to have hundreds of thousands of tests in order to test every nitty-gritty detail. And it would also be extremely brittle as something changes over time."

The suite answers "did we break the old things we thought to test." It can't say whether the new thing works, or whether the change broke something nobody wrote a test for.

Regression testing in the pull request: build, scope, verdict

The schedule is the least of what changes when regression testing moves onto the PR. The build, the scope and the verdict all move with it.

The build. The tests run against the PR's own deployment, before the code reaches main. On Vercel this is the default: a preview deployment is created when you "create a pull request (PR) on GitHub, GitLab, or Bitbucket", with its own URL posted back into the PR. Other hosts and custom CI can do the same through GitHub Deployments. One QA manager we spoke to is mid-move. PRs used to merge to a dev branch that got deployed and tested after the fact; now, "we're switching to PR-based instances. So as the PR is made, there's a specific instance just for that."

Without a per-PR build there is no pre-merge regression testing, whatever a tool's landing page says. The fallback is to run the same scoped review after merge, against staging, and post the result back on the merged PR. That catches the regression hours earlier than the nightly run would, and nothing stops the merge.

The scope. Google has run only the tests a change can affect since at least 2011, when its CI used dependency analysis to pick, per change, "all the tests a change transitively affects." Facebook's predictive test selection halved testing infrastructure cost while still reporting "over 99.9% of faulty changes" to developers. A PR that only touches the CSV export has no reason to re-run password reset.

Where selection falls short is the same place the nightly suite does. It can only run what exists. If the PR adds a flow, or changes behaviour no test covers, selection picks nothing and the check goes green. Scoping to the change has to mean both: pick the existing tests the diff can affect, and write the ones it needs that aren't there yet.

The verdict. A nightly red is an email. A PR verdict is a status check, and if it's required on a protected branch "they must pass before the pull request can be merged." That changes what the verdict has to be able to say. Pass and fail don't cover it. When the preview never came up or the login fixture broke, nothing exercised the change at all. A tool that reports that as a pass is a gate that opens for the wrong reason, and GitHub already ships one: a skipped job reports "Success" and does not stop a merge even as a required check. The tool should say unable to verify, and a human reads why.

One more thing if you're going to gate. Two PRs can each pass their own checks and still break main together. GitLab's merge trains exist for exactly this: "two merge requests can each pass their own pipeline, but their combined changes can still conflict." GitHub's merge queue does the same job and needs your CI to run on merge_group events. If PR checks are what keeps main green, they have to run on the merged result as well as the branch.

What goes on the PR, and what stays nightly

The question we get is whether the PR gate replaces the regression suite. It doesn't. The teams who've thought about it hardest run both, with different jobs.

An engineering manager on a twelve-year-old product has Selenium running the full regression every Friday and isn't planning to replace it. What he wanted on the PR was narrower: "I'm really more focused on just that PR, just that change versus everything across the application from a regression perspective." His other line explains why code review on its own isn't the answer: "we do code reviews, but obviously that's not going to catch major issues or things that are underlying issues that you can't see by a code review."

Daniel Mauno Pettersson, our co-founder and CEO, describes the layering to customers in three parts. First, the flows that must never break: "there you have your 20, 50, 100 tests. They are hugely important, they must never break, and they are always green." Second, the change in front of you: "the new stuff, that is maybe more a judgment call for you: do we ship this even though this didn't work? Because maybe it is not that big a deal. That is more exploratory testing." Third, the fast coded tests the developers own, which "maybe run all the time and are really fast. But they usually only cover the happy path."

So the PR gate is the small always-green suite plus the tests scoped to the diff. Everything else stays on a schedule, where a red costs someone an hour and holds up no merge. Google's presubmit has the same shape, even though theirs is "often a project's unit tests": a fast subset runs before submit, deliberately excluding slow and unreliable tests, and a change that passes it has "a very high likelihood (95%+) of passing the rest of the tests." The average wait to submit is around 11 minutes.

If the always-green suite is currently four hundred tests, the problem isn't where you run it. We've written about why a large regression suite is a liability. The PR gate should be small enough that a red is always worth reading.

AI regression testing in the pull request: selecting tests versus generating them

The phrase "AI-driven regression testing in the pull request workflow" covers two different products, and the difference is the thing to check before you buy either.

The first kind selects. It reads the diff, maps it to the tests you already have, and runs that subset against the preview. Momentic's PR testing does this: it "runs only the tests that AI test selection picks from the diff", triggered when the preview deployment is marked ready, with the result as a required check. Others in this category skip the selection and run the whole existing suite in parallel on each preview, or run the plans you've already built on a deployment event and let you require the check. All of them are clear about what they are: your tests, chosen or not, on the PR's build. None of them writes a test inside the PR run. Where a new test comes from a coding-agent skill, it lands in your working tree and goes through the same review. (Checked on each vendor's page, 20 Oct 2026.)

The second kind generates. It reads the change, and sometimes the ticket, works out which user-facing behaviour the PR touches, picks the existing tests that cover it, and writes the ones that don't exist, then runs them. A few tools now do this from a PR comment or a PR event, and so does QA.tech, an AI testing platform that integrates directly into the GitHub pull request workflow through dynamic testing on the pull request. It won't catch what it can't reach: it tests through the UI or an API, so a backend-only change with nothing user-facing comes back as unable to verify. What it removes is the guesswork about which flows a diff can break.

When a pull request opens, the review agent reads, in order, the PR description, any linked Jira or Linear ticket including its acceptance criteria, the changed files, and the commit messages (docs). It classifies the change, finds the existing tests that cover the affected flows, scores the risk from blast radius, how central the flow is and what's already covered, and sets the depth from that. The whole review runs autonomously, with nobody starting it or picking the tests (docs). There is no fixed number of tests per PR. A docs-only change is skipped, with the reason on the check, a small change to checkout can be high risk, and the docs' own line is that most bug fixes and refactors create no new tests at all. Where coverage is missing it creates test cases for the gaps, runs everything against the PR's preview, and posts pass, fail or unable to verify with a per-test results table on the PR; screenshots, the recording and the step trace are one click away in the run. The check is called QA.tech / PR Review. Require it in branch protection, or set blocking: true on the GitHub Action so the step waits for the verdict and fails when the review ends in FAILED or CANCELLED (docs). After merge on GitHub, a post-merge agent can promote test cases the review created into the regression suite, which activates them and labels them regression and auto-added; it promotes only what its prompt asks for, and promoting nothing is a normal outcome (docs).

The limits are the ones the three differences predict. It needs a preview URL to test the PR's build; without one, the automatic review has no PR-specific URL and ends up running against your default environment, which is the wrong build (docs), so the documented pattern is to run the review after merge against staging instead. The review gates on GitHub only; on GitLab it posts a note and doesn't fail the pipeline. The one way to block a GitLab merge is a test plan run from CI in blocking mode, which runs a fixed list you chose.

Five questions before you gate a merge on it

Does it run the application, or only read the code? Review bots read the diff. As Vilhelm put it on the same webinar, "the real truth happens in the end result of the product, when everything comes together", and a regression is only visible there.

Does it choose tests from the change, or run a fixed list? And if it chooses, can it also write the test the change needs that you don't have?

What does it return when it couldn't exercise the change? If the answer is "green", it isn't a gate.

Does it need a preview environment, and what happens without one? The acceptable fallback is an after-merge run on staging, posted back to the PR.

Can it block the merge, and on which host? Required status checks are a GitHub mechanic; a blocking step is whatever the vendor's Action offers. Check the equivalent exists where your code lives.

FAQ

What is regression testing in a pull request? Running the checks that would catch a regression against the pull request's own build, before merge, scoped to what the change can affect, and reporting the result as a status check the merge can wait on. It differs from a nightly regression run on three points: the build under test is the PR preview, the scope is what the diff can affect, and the verdict can block the merge.

Should regression tests run on every pull request? A small set should: the flows that must never break, plus tests scoped to what the PR changed. The full suite should not. Running everything on every PR is slow, and at Google's scale 14% of large tests were flaky, so a full run on every change produces reds that have nothing to do with the change. Keep the full run on a schedule.

What is the difference between pre-merge and nightly regression testing? Pre-merge testing runs on the PR's preview deployment and can stop the merge. Nightly testing runs on main after many changes have landed and can only tell you that something in the last day broke. A nightly red needs a bisection; a pre-merge red names the PR.

Can AI select which regression tests to run on a pull request? Yes, and the technique predates the current wave: Google has selected tests by change dependency since 2011, and Facebook's predictive test selection halves test cost while reporting over 99.9% of faulty changes. The limit of selection is that it can only run tests that exist. Catching a regression in behaviour nobody wrote a test for needs the tests generated from the change.

Do I need a preview environment for pull request regression testing? To test before merge, yes. The tests have to run against a build that contains the change and nothing else that hasn't merged. Vercel, Netlify and similar hosts create one per PR; custom pipelines can register a GitHub Deployment. Without one, run the scoped review after merge against staging and post the result back to the PR.

Which tools run regression tests on pull requests? Two kinds. Tools that select or run your existing tests on the PR's preview, and tools that read the change and generate the tests it needs as well as selecting existing ones. Check which kind you are looking at, whether it can report that it could not verify the change, and whether it can block a merge on your Git host.

Can I drop the nightly regression run if every pull request is tested? No. Keep a scheduled suite for the flows that must never break, and put the tests scoped to the change on the pull request. A PR gate can't see drift that builds up over many merges, and a nightly run can't say which merge caused a red, so neither does the other's job.