AI Testing in CI/CD: How to Run E2E Tests on Every Pull Request

Most teams don't have a testing problem so much as a placement problem. The tests exist, they run somewhere, and the breakage still reaches production, because the place they run isn't the place where the decision to ship gets made.

Where tests should run, and what each position buys you

Testing has been the slow part of software delivery for years, and for most of those years that was tolerable, because writing the code was slow too. That stopped being true.

“If now coding is so much faster, by the simple theory of constraints, which is an old production line thing that has been around for ages, a system is never faster than its slowest part. And if we then shift the bottleneck from development to later in the pipeline, testing quickly becomes the new bottleneck instead. And the only way to solve this and improve the overall throughput of the system is to automate the slowest part.” – Vilhelm von Ehrenheim, QA.tech

Daniel Mauno Pettersson, QA.tech's CEO, puts the same point from the release side: teams used to accept a two-week release cycle, agentic coding changed the expectation, and the pressure landed straight on verification.

So the bottleneck moved to the pull request, which is also where the four available positions for a test differ most.

Position	Catches	Costs	Who it's for
Local, before push	Obvious breakage in what you just wrote	Developer time and discipline; nothing enforces it	Everyone, and it's never enough on its own
On the pull request	Regressions in the change's blast radius, before anyone else is affected	Minutes of CI per PR; needs a preview environment	The default position for end-to-end tests
On merge to main	Interaction bugs between changes that were fine separately	Found after the fact; the author has moved on	A backstop, not a substitute
Before release	Whole-product regression, run as a regression suite; anything the earlier layers missed	Slow, and by now expensive to fix	The flows you'd cancel a release over

The argument for the pull request as the default is short: you should verify on the PR, because it no longer makes sense not to. The failure mode of relying on the merge instead is one a prospect described better than we could:

“One of the things we are lacking massively is when something happens, you fix this. But it could be impacting in a completely different part of the system. And we don't know it until it's shipped and people, we get support requests.” – a CPO at a US healthcare-software company

Which flows belong at which position is the first of the four decisions in our guide to software testing strategies.

Setting up test automation in GitHub Actions

The mechanics first, because the rest of this article assumes them. On GitLab the shape is the same and the syntax isn't – we compared GitLab CI and GitHub Actions for this specifically.

A workflow is a YAML file in .github/workflows/. It runs one or more jobs when an event fires. Jobs run in parallel by default, each on a fresh runner, and each job is a series of steps: a shell command, or an action someone else already wrote.

The smallest useful version:

# .github/workflows/test.yml
name: Node.js CI

on:
  push:
    branches: [ "main" ]
  pull_request:
    branches: [ "main" ]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'
      - name: Install dependencies
        run: npm ci
      - name: Run tests
        run: npm test

The on block is where most of the placement decision actually gets made. Three common shapes:

# Every push and PR: most feedback, most CI minutes
on:
  push:
    branches: [ "main" ]
  pull_request:
    branches: [ "main" ]

# PRs only: the usual choice once CI minutes matter
on:
  pull_request:
    branches: [ "main" ]

# Path-filtered: for larger repos where most changes touch neither
on:
  pull_request:
    branches: [ "main" ]
    paths:
      - 'src/**'
      - 'tests/**'

Unit tests belong here, in the repository, running on the runner. End-to-end tests are the ones that need somewhere to point a browser at, which is the next section and the part teams get wrong.

Triggering tests from a pull request

The hard part of pull-request testing isn't the test. It's that the thing you want to test has to exist before you can point a browser at it.

Most teams hit this the same way. The workflow triggers on pull_request, starts immediately, the preview deployment is still building, the tests hit a URL that isn't serving yet, and everything goes red for reasons that have nothing to do with the change. Then somebody adds a sleep 60, which works until it doesn't.

Before any of the mechanics: if you don't have per-PR preview environments, sort that out first. It is the actual prerequisite, and in our experience it's the single biggest predictor of whether pull-request testing sticks. Teams that try to bolt it onto a shared staging environment end up serialising their pipeline and chasing false failures from two changes colliding. There's a post-merge pattern further down that works in the meantime, but it isn't the same thing.

Two ways in, and they answer different questions.

	Autonomous review	Test plan run
What it does	The agent reads the diff, picks the relevant flows, fills coverage gaps, runs them, posts a review on the PR	Runs a test plan you defined, against a URL you specify
Who chooses the tests	The agent	You
How you trigger it	GitHub App automatically, or the Change Review Action from CI	Test Run Action, or the API from any CI system
Use it for	Every pull request, where you want coverage you didn't have to plan	Regression suites, nightly runs, deployment gates

Most teams end up running both: the autonomous review on every pull request, and a named test plan as the deterministic gate before release.

Option A: let the GitHub App do it. Install the App at the organisation level, map the repository, and reviews start appearing on pull requests with no workflow file at all. The App is also what ingests pull-request data in the first place, so it's the foundation either way rather than an alternative to the Actions. The full GitHub App configuration, including the permissions it needs and how the check run is registered, is in our docs.

Option B: drive the review from CI. Reach for the Change Review Action when the App's automatic trigger isn't flexible enough. The common reasons: more than one application per pull request, so the App's environment mapping gets fiddly; or you need the review to fire after a specific deploy job rather than on PR open.

name: QA.tech Change Review

on:
  pull_request:

jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - uses: QAdottech/run-action/change-review@v3
        with:
          api_token: ${{ secrets.QATECH_API_TOKEN }}
          blocking: true
          applications_config: |
            {
              "applications": {
                "YOUR_APP_SHORT_ID": {
                  "environment": {
                    "url": "https://preview-${{ github.event.number }}.example.com"
                  }
                }
              }
            }

Three things in that file are worth understanding rather than copying.

applications_config is keyed by application short ID, not by a name you invent. Find yours under Test Plans → API Integration. Projects with several applications pass one entry each, which is the whole reason to use this action instead of the App.

blocking: true is what turns this into a gate. Without it the step returns as soon as the agent starts working. With it, the step waits and fails on FAILED or CANCELLED, so a branch-protection rule has something real to depend on. It polls every 20 seconds and gives up after 60 minutes, so set a tighter timeout-minutes if you want a shorter leash.

You don't need a project_id. The API token already scopes the request to a project.

Waiting for the preview. Do it with job dependencies, not a sleep. The deploy job publishes the URL, the test job consumes it.

jobs:
  deploy:
    runs-on: ubuntu-latest
    outputs:
      preview_url: ${{ steps.deploy.outputs.url }}
    steps:
      - name: Deploy preview
        id: deploy
        run: |
          # your deploy, then publish the URL it produced
          echo "url=https://preview-${{ github.event.pull_request.number }}.example.com" >> "$GITHUB_OUTPUT"

  test:
    needs: deploy
    runs-on: ubuntu-latest
    steps:
      - uses: QAdottech/run-action@v3
        with:
          api_token: ${{ secrets.QATECH_API_TOKEN }}
          test_plan_short_id: 'regression-suite'
          blocking: true
          applications_config: |
            {
              "applications": {
                "app_frontend": {
                  "environment": {
                    "url": "${{ needs.deploy.outputs.preview_url }}",
                    "name": "PR-${{ github.event.pull_request.number }}"
                  }
                }
              }
            }

The thing that catches everyone: protected previews. Vercel and Netlify put previews behind authentication by default, so an agent gets a login wall instead of your application and every flow fails for the same uninformative reason. Pass the bypass as a header rule on the environment:

            {
              "applications": {
                "app_frontend": {
                  "environment": {
                    "url": "${{ needs.deploy.outputs.preview_url }}",
                    "customHeaders": [
                      {
                        "domains": ["*.vercel.app"],
                        "headers": {
                          "x-vercel-protection-bypass": "${{ secrets.VERCEL_AUTOMATION_BYPASS_SECRET }}",
                          "x-vercel-set-bypass-cookie": "true"
                        }
                      }
                    ]
                  }
                }
              }
            }

customHeaders persists onto the environment. Omit it and stored headers stay as they are; pass [] to clear them.

No preview environments yet? Run the review after the merge instead. It's worse, because you find out once the change is already on main, and it is much better than nothing. Deploy to staging on push, pull the PR number out of the merge commit, and review the merged pull request against staging. Results land as a comment on the closed PR.

      - name: Extract PR number from merge commit
        id: pr
        env:
          COMMIT_MSG: ${{ github.event.head_commit.message }}
        run: |
          PR_NUMBER=$(echo "$COMMIT_MSG" | grep -oE '\(#[0-9]+\)' | tail -1 | grep -oE '[0-9]+')
          if [ -n "$PR_NUMBER" ]; then
            echo "url=https://github.com/${{ github.repository }}/pull/$PR_NUMBER" >> "$GITHUB_OUTPUT"
            echo "found=true" >> "$GITHUB_OUTPUT"
          fi

Direct pushes with no pull request skip themselves, because the review step is gated on steps.pr.outputs.found.

Credentials. One secret, QATECH_API_TOKEN, from your project's integration settings. Add it under Settings > Secrets and variables > Actions and never inline it in a workflow file.

Reading the result. The Test Run Action exposes run_result as PASSED, FAILED or SKIPPED, and run_status as COMPLETED, ERROR, CANCELLED or TIMED_OUT, both only when blocking. The Change Review Action exposes chat_status and chat_url. Use chat_url when you want to link a human to the conversation; the review itself is already on the pull request, posted by the agent, so don't build a second comment that duplicates it.

On other CI systems: the API is plain REST, so anything that can make an HTTP request can trigger a run – GitLab, Bitbucket, CircleCI, Jenkins, Azure DevOps. Native merge-request reviews with a status check exist for GitHub and GitLab. On the others you can trigger and poll, but the verdict won't appear as a native check.

Gating a merge on the result

Most CI-testing writing says "add it as a required check" and stops, which skips the actual problem. The interesting problem is trust, not configuration.

Teams want a machine verdict and a human decision point, not one replacing the other:

“In a perfect world, a ticket goes to QA ready, and then it comes to QA passed, and then it just needs a human gate to go: am I happy with that? For me, the intervention is you put human gates in the way at certain key points, so it can't move past this without a human going yes or no.” – a founder at a UK hospitality-software company

“It's not a confidence score – whatever that is. I just want to know if it comes through QA.tech and it goes, yeah, great.” – same speaker

Nobody wants a percentage. They want a verdict. What makes a verdict trustworthy is narrow and boring: a trace an engineer can read, a re-run policy so one red doesn't block on its own, and a published real-red rate. A gate that fires for reasons nobody can explain is how teams learn to ignore it, which is the practical difference between the two kinds of flaky tests – the ones that mean something and the ones that cost you the gate.

There's also a reason a machine verdict earns its place at all, and it isn't about speed:

“If you let developers do this, they inherently don't necessarily think about all the things that they could potentially break. There is some kind of inherent bias in letting developers test their own stuff.” – Vilhelm von Ehrenheim, QA.tech

The recommendation: gate on a small named set of critical flows from day one, report everything else without gating, and move flows into the gated set as their real-red rate earns it.

When the tests are run by an agent instead of a script

Same pipeline, different thing inside it. A scripted suite needs coded steps and a decision, made in advance, about which tests apply to which change. An agent gets a goal in plain language and the diff, and works out the rest per run.

The published shape of the review is four steps: classify changes – docs-only or infrastructure-only changes skip testing and post an info comment; assess coverage – find relevant existing tests, identify gaps; create tests for the gaps, configuring dependencies such as login; run them against the environment. Typically 5 to 15 tests get selected per review, scoped to what the change touched rather than to the whole product, which is what dynamic testing means in a pipeline.

The trade is real and worth stating plainly: runs are not byte-identical. An agent that decides its own steps will take a slightly different path on Tuesday than it took on Monday. For most end-to-end coverage that's fine, and it's why selector maintenance disappears. For the handful of checks that genuinely must be identical every time – a pricing calculation, a signed payload – keep the script.

Where results should land

Wherever the decision gets made, and nowhere else. A POC sponsor read us his own success criteria and they're the whole section:

“Visibility and reporting: results visible in PR checks, plus Slack, within 10 minutes of run completion. Failure reports actionable without opening the QA.tech UI.” – a POC sponsor at a European software company

“I want a tool that is going to be deeply embedded into our CI/CD… I want fast feedback on all merge requests, where instantly a developer can say, yep, that looks good. It needs to be a tool that's integrated into a wider ecosystem.” – an engineering leader at a US telehealth company

In practice that means the check run and a PR comment as the primary surface, Slack for the people who aren't watching the PR, and a ticket in Jira or Linear only when something needs to become work. Another dashboard is not a reporting strategy.

Where QA.tech fits

QA.tech runs the agent side of this: PR testing on every pull request, against your preview environment, with the verdict posted as a review before merge. You describe flows as goals rather than steps, and the agent decides how to reach them on each run.

The configuration itself lives in our docs rather than in this article, because that's the copy that stays current. What's worth knowing here: it needs a reachable interface, so a service with no frontend isn't testable this way; the runs aren't byte-identical, as above; and as any agentic tool it won't catch everything – our own founders put it at around 90% of the routine verification, which is useful precisely because it's the part nobody wants to do by hand.

Best practices for a robust setup

1. Use secrets for sensitive data. API tokens, passwords and signing keys belong in GitHub secrets, never in a workflow file.
2. Make workflows debuggable. Clear logging and uploaded artifacts are what turn a red run into a five-minute fix rather than an afternoon.
3. Separate workflows by speed. unit-tests.yml for fast checks, a separate workflow for the heavier end-to-end testing with AI agents, so a slow suite never blocks a quick one.
4. Cache dependencies. actions/cache on node_modules pays for itself immediately on any repo with a real dependency tree.
5. Keep the PR run under ten minutes. That's the number where people wait for the result instead of merging around it.

If you want the agent side running on your own pull requests, get a demo.

QA.tech also exposes an MCP server so coding agents can trigger tests directly.

Related reading

• Dynamic PR Testing with AI Agents
• Mobile Testing
• QA Agents for E2E Ecommerce Testing
• Automating QA with AI in GitLab CI/CD: A Step-by-Step Guide
• Automate Testing in Your CI/CD Pipeline with QA.tech
• From Manual to Autonomous QA: Step-by-Step Guide