# How Does an AI Agent Know What to Test on a Pull Request?

> Is an AI agent just guessing what to test on a pull request? See what it reads (PR description, ticket, diff, test cases, app map) and what each one misses.

Source: https://qa.tech/blog/how-ai-decides-what-to-test · Published: 2026-11-16

---
An AI testing agent works out what to test on a pull request from evidence it can point to: the PR description, linked ticket, diff, existing test cases, a map of the app and your team's rules. It runs the test cases that cover the affected flows and writes new ones only for gaps. Thin inputs push it toward guesswork, so a usable agent says what it tested and why.

## How an AI agent decides what to test on a pull request: is it just guessing?

The doubt usually comes as one word: guessing. You hand an agent a diff, it picks a few flows, and nobody can say why those ones.

It's a fair thing to push on.

### What guessing would look like

An agent that's guessing has a diff, a URL and nothing else. It spots `CheckoutForm.tsx` among the changed files, opens the app and clicks around whatever looks like checkout. It doesn't know the same form sits inside account settings. It can't tell whether the button that moved was the point of the change or an accident. Run it again tomorrow and it picks different flows, and on neither day can it give you its reasons.

Some setups do work this way. A general-purpose agent pointed at a browser and told to "test this PR" comes close.

What separates inference from guessing is what the agent has to go on, and research on coding agents shows how much that matters. In [TDAD, a March 2026 arXiv preprint](https://arxiv.org/abs/2603.17973), handing a coding agent a map from source code to tests cut the regressions it introduced by 70% (6.08% to 1.82%), while telling it to work test-first without that map pushed regressions up to 9.94%. It's a small evaluation on one benchmark, so don't lean on the exact figures. The direction is what's worth keeping. The agent that could look up which tests a change touched broke less than the one that was only told how to work.

### What the agent has to go on

"A diff is not enough to really understand what the potential risk is," says [Vilhelm von Ehrenheim](https://www.linkedin.com/in/vilhelm-von-ehrenheim/), our co-founder and Chief AI Officer.

A person reviewing the PR by hand wouldn't stop at the diff either. They'd read what the author wrote, open the ticket and lean on what they know about the product. An agent built for pull requests gets the same material. Ours [ranks the context signals in this order](https://docs.qa.tech/best-practices/pr-testing):

1. **The PR description.** What the author says the change does. It's the strongest signal there is.
2. **The linked ticket.** With the Jira or Linear integration connected, put the ticket key in the branch name or a commit and the agent fetches the ticket, including its [acceptance criteria](https://qa.tech/blog/acceptance-criteria-agentic-testers).
3. **The changed files and the diff.** Which components, routes and handlers were touched.
4. **The commit messages.** The change told in steps, late fixes included.

Those four give it intent and scope. The rest turns that into a list of things to run.

**The test cases you already have.** The agent reads them all and matches the change to what each test case is for. The match is on meaning, so a PR that reworks the payment step pulls in "complete checkout with a saved card" even when the two share no keywords. Test cases that merely live in the same application stay out. Picking from tests that already exist is an old problem, and the code-map and prediction approaches to it go by the name [test impact analysis](https://qa.tech/blog/test-impact-analysis).

**A crawled map of the app.** [The knowledge graph](https://docs.qa.tech/core-concepts/knowledge-graph) is built by walking the product the way a user would, with the logins you gave it. It holds the pages and the interactions between them, so a changed route or screen can be tied to the journeys that pass through it.

**The source code, as a fallback.** When the diff and the map don't settle it, the agent can clone the repository and search for where the changed code is used. A three-line edit to a shared date picker looks small until that search finds it in booking, invoices and the reports filter. A review reads the description, the diff and the commit messages. Cloning the repository to search the rest of the code is a separate setting, [code access](https://docs.qa.tech/configuration/github-app#code-access), which is on by default for newly connected repositories and can be switched off.

**Rules your team wrote.** [PR Review Rules](https://docs.qa.tech/core-concepts/knowledge#rules) are standing instructions in plain language: which account to log in with for anything under billing, which app in the monorepo a folder belongs to. On GitHub and Azure DevOps, a one-off instruction can go in a comment on the PR.

People often ask whether it also learns from earlier pull requests. What carries over is more ordinary than learning: a test case written for one PR can be reused when a later one touches the same flow, and the rules your team writes stay in force.

All of this is about choosing. The testing itself happens in the running app, where the agent opens the PR's [preview environment](https://qa.tech/blog/preview-environments) in a browser and goes through each flow as a user would.

### Each input, and what it misses

| Input | What it tells the agent | What it misses |
|---|---|---|
| PR description | What the author meant to change and what they think needs testing | Everything the author left out. An empty description leaves the diff as the only statement of intent |
| Linked ticket | The acceptance criteria and the reason for the change | Criteria kept in Jira comments or custom fields, and anything agreed in a call |
| Changed files and diff | Which components, routes and handlers were touched | Whether a behaviour change was intended, and where else that code is used |
| Commit messages | The order things happened in, including late fixes | User-facing behaviour, often ("fix lint", "wip") |
| Source code (fallback) | Where the changed code is used across the repository | Configuration and data that live outside the repository |
| Existing test cases | Which flows are already covered and how your team describes what matters | Flows nobody has written a test case for yet |
| Crawled map of the app | Which pages and interactions exist and how a user reaches them | Screens behind a login or role it was never given |
| Team-written rules | What only your team knows: accounts, exceptions, flows that always matter | Whatever nobody thought to write down |

## How risk sets the depth

One test, five, a thousand? There isn't a magic number, and a tool that runs the same amount on every PR isn't looking at the change.

Depth follows risk. The agent weighs the blast radius, meaning how many user flows the change can reach. It weighs how central those flows are (login, sign-up and payments get more scrutiny), and how much of the affected area already has passing test cases. Diff size on its own decides nothing. A one-line change to checkout can come out high. A sprawling change to an isolated, well-covered settings page can come out low.

The result is a level, low, medium or high, the tiers set out in [risk-based testing for pull requests](https://qa.tech/blog/risk-based-testing-pull-requests). There's a fourth outcome, "not assessed", for when the diff and what's known about the project are too thin to say which flows are affected. That's the agent telling you it doesn't have enough to go on.

Whatever the level, any user-facing functional change gets its success path tested directly. At high risk, existing smoke test cases for the main and neighbouring journeys are added on top. The level only sets how deep the testing goes. It doesn't bend how pass and fail are decided.

## When it writes a new test case, and when it reuses one

Reuse comes first. Because the agent has read your existing test cases, it knows what's covered before it writes anything. That's also what stops it producing duplicates.

A new test case shows up only where there's a gap. Say a PR adds "pay by invoice" to checkout. Card and wallet checkout already have test cases, so those run. Invoice has none, so one gets written. How many get written depends on how many breakable paths are uncovered and how risky the change is. Most bug fixes and refactors add none.

Bug fixes get one extra move where the setup allows it. Given a usable base environment and a bug that can be reproduced end to end, the review can try to show the failure on the base build first and the fix on the preview second.

Test cases written during a review start life as drafts, kept out of your main test list. On GitHub, push another commit and the review carries on in the same conversation. The agent can rerun the test cases the new commits affect, and change or drop the ones it wrote for this PR if they no longer match.

## What triage skips

Before any of that, a cheaper question gets asked: whether there's anything here to test at all.

A README edit, a CI config tweak, an infrastructure manifest, a change to build tooling. None of those has behaviour an end-to-end test could exercise, so [a triage step](https://docs.qa.tech/pr-testing/pr-triage) skips them. It sees the PR's title, description, branches and changed file paths, and nothing inside the files, so think of it as a fast, coarse filter in front of the expensive part. When it skips a pull request on GitHub the check says why, and a comment that mentions the agent runs the review anyway.

## What the agent cannot know

Some gaps stay open until a person closes them.

**Intent nobody wrote down.** Give it a PR titled "payment stuff" with an empty description and it infers intent from the diff alone. [Our own docs](https://docs.qa.tech/best-practices/pr-testing) call the result "more guesswork", and that's the right word for it. Say the diff flips the default sort on the invoices list. The list still loads and every row is there. If the description or the ticket says which order was wanted, the agent can check it. If neither does, it has to pick a reading, and it can pick the wrong one.

**Flows it can't reach.** The agent sees what a user with its logins sees. A feature behind a flag that's off in the preview, a screen only an auditor role can open, a step guarded by a CAPTCHA, a report that needs six months of data the preview doesn't have. All of these are out of reach until someone gives it a way in. Each role needs its own saved account. A flag needs a header or a rule that says when to switch it on. And when it couldn't confirm the change, the verdict is "unable to verify", so you hear about it.

**Business rules that live in someone's head.** Refunds over a certain amount need a second approver. Trial accounts never see the export button. Neither is written down, because everybody on the team knows. The agent will confirm that the refund went through and move on. Once the rule is in a ticket or a written rule, it's an input like any other.

A few narrower limits belong here too. The agent has no opinion on code quality, so a bug shows up only when a test reproduces it. Two reviews of the same change can pick slightly different test cases, and written rules pull later runs toward the same guidance without making them identical. A backend change is tested through the flows that use it.

None of this says how far to believe a result once the tests have run, which is what [can you trust an AI agent to run your tests](https://qa.tech/blog/trust-ai-agent-to-run-tests) is about.

## How to check the selection

You don't have to take the selection on faith, because a review that's any use states what it tested and why. Look for these on the pull request:

- the risk level and the reasoning behind it
- a description of what was covered
- a table of the individual test cases with their results
- a verdict: pass, fail or unable to verify

Checking takes a minute. Does the risk line match what you know about the change? Is the flow you were worried about on the list? If it isn't, mention the agent in a comment on the PR, something like "also test the refund flow as an admin", and it runs the review again with that instruction. If the same miss turns up twice, write a rule.

The evidence behind each result (screenshots, logs, a trace, and a recording when one was captured) sits in the run, linked from the PR.

For the few flows that must be checked on every change, don't leave it to selection. Give them a short fixed suite, one of [the checks that belong on every pull request](https://qa.tech/blog/pull-request-checklist), and leave the rest to the agent's context-based testing of the change.

## What our PR review does with these inputs

Our review has nothing to say about your implementation, so code review stays where it is. It doesn't run byte-identical steps every time, so anything that must be exactly repeatable belongs in a script. And it can't test a flow it has no way into.

QA.tech, an [AI testing platform](https://qa.tech/), reviews pull requests with the inputs described above. It reads the description, the linked ticket, the diff and the commits, matches them against your test cases and its map of your app, runs the affected flows on the preview deployment, and posts a verdict of pass, fail or unable to verify next to the risk level and the list of what it tested. Whoever approves the PR can see what was exercised, and that's what lets them merge with confidence. We call the approach [dynamic testing](https://qa.tech/blog/what-is-dynamic-testing), and in the product it's [dynamic PR testing](https://qa.tech/product/pr-testing).

Any of the AI-driven automated testing platforms that integrate directly into GitHub pull request workflows will post a result on the PR. Ask each one what it read to get there, and whether it will show you the selection.

On GitHub you can make the review a required check, so nothing merges on a failed verdict. On GitLab and Azure DevOps the review posts on the request. The fixed suite from the last section is a test plan you run from CI on every pull request, alongside the review. To steer a review on GitHub, comment something like `@qa.tech test the invoice flow as a finance user`.

The free trial on the [pricing page](https://qa.tech/pricing) covers one application: connect a repository, open a pull request and read what it picked and why.

## Give it less to guess

The cheapest improvements are all on your side of the PR.

Write the description for a tester who has never seen the feature: what the user will notice, the reason for the change, the flows you'd want tried and anything the PR deliberately leaves alone. If time is short, have the coding agent that wrote the change draft it. Put the ticket key in the branch name, and keep the acceptance criteria in the ticket's description where they'll be read. Save a login for each role that matters. Write down how a flag gets switched on.

Everything here happens before the merge. Afterwards the question is which of these test cases you keep, and that's [how PR test cases become your regression suite](https://qa.tech/blog/pr-test-cases-regression-suite).

## FAQ

### How does an AI QA agent decide what to test?

It reads what the change is meant to do (the PR description and linked ticket), what it touched (changed files, diff, commits), and what the team already has (existing test cases, a map of the app, written rules). From those it works out which user flows the change can reach, runs the test cases that cover them and writes new ones for the gaps. How deep it goes depends on risk.

### Is an AI testing agent just guessing?

With only a diff and a URL, close to it. With the author's stated intent, the ticket, existing test cases and a map of the application, it's making an inference you can inspect, much as a human reviewer does. The test is whether the agent reports its selection and reasoning. If it can't say what it tested and why, treat the result as a guess.

### How do I generate an end-to-end test from a pull request diff?

Don't start from the diff alone. A diff says which code changed and very little about what should now be true for a user. Pair it with intent: the PR description or the ticket's acceptance criteria. Give the agent your existing tests as well. In [ChaCo, a January 2026 arXiv preprint on LLM-written tests for pull requests](https://arxiv.org/abs/2601.10942), run on three open-source Python projects (library code, so not end-to-end tests), the generated tests added 2x the coverage when the model was given the project's existing tests as context. Then run the generated test against a deployed build of the branch.

### How do I run only the end-to-end tests that a pull request changed?

The static route is path filters: map directories to test tags and have CI run the tagged subset when files under those paths change. It's cheap, and shared components defeat it. The other route is selection by user flow, where an agent matches the change to the flows it can reach and runs the test cases covering them. Either way, keep a small fixed set of critical-path tests running on every pull request.

### What is predictive test selection and how does it work?

Predictive test selection uses a model trained on past code changes and test outcomes to pick the tests most likely to fail for a new change. For each test it estimates how likely that test is to catch a regression in this particular change, then runs the likely ones and skips the rest. It needs a history of test results to learn from. Coverage-based selection works from a map of code to tests instead, and both struggle with end-to-end tests for the reasons in our guide to [test impact analysis and predictive test selection](https://qa.tech/blog/test-impact-analysis).

### What can an AI testing agent not know about a pull request?

Intent that nobody wrote down, flows it has no login, role or flag setting to reach, and business rules that exist only in people's heads. It also can't judge code quality by running the app. Most of these shrink once the missing piece is written into the PR description, the ticket or a standing rule.
