Quality Assurance·

How to Write Acceptance Criteria for Agentic Testers

How to write acceptance criteria an AI testing agent can check in a running app: five rules, four before-and-after rewrites and a PR description template.

QA.tech team

Acceptance Criteria – a precision brass measuring gauge checking a frosted acrylic component against a glass template.

Acceptance criteria for an agentic tester are statements an AI agent can check in the running app: who is logged in, where they start, what data to use and what the user should see. Criteria written for a teammate fail because they lean on context the team carries in its head. Write one per user outcome, add a case that should fail, and say what's out of scope.

Why acceptance criteria written for people fail an agentic tester

Take a ticket that says "User can export invoices." A teammate reads that and knows which user, which page, which invoices and roughly what the file should contain. An agent opening the preview build has to work it out from the diff and whatever it can see in the app. It'll still test something. Whether that something is what you meant depends on what you wrote down.

Atlassian describes acceptance criteria as "a set of clear, concise, and testable statements that focus on providing positive customer results". Testable is the word that matters. Bill Wake said the same when he set out the INVEST checklist in 2003: writing a story card carries "an implicit promise: 'I understand what I want well enough that I could write a test for it.'"

In practice that promise gets kept later, by a person, from memory. Some teams have no documented test cases at all. Someone who knows the product reads the card and knows what to try. Every senior person on a team carries a long list of unwritten rules, and the criteria can stay thin because the reader fills them in.

That arrangement breaks the day that person leaves. It also breaks when the reader is an agent, and that now happens at both ends of a ticket: teams point a coding agent at the tracker and tell it to go fix the issue, then a testing agent reads the same lines to check the result.

The same gaps keep showing up.

GapA line you'd find in a real ticketWhat the agent has to guess
Ambiguity"Checkout should handle payment errors gracefully"Which errors, and what "gracefully" looks like on screen
Missing start state"User can cancel their subscription"Which plan, which role, before or after the renewal date
Missing data"Admin can bulk-import contacts"Which file, how many rows, what a bad row looks like
No observable outcome"Preferences are synced to the backend"What a user would see if it worked

None of these lines is wrong. Each is a summary of a conversation the agent wasn't in.

An agent handed a vague line doesn't stop and ask. It infers intent from the diff and tests what it can reach, and that's where the worry that it's only guessing comes from. Written criteria are how you take the guessing out. They're still only intent, though, and nothing is checked until something opens the app and tries each one.

Five rules for acceptance criteria an agent can check in a running app

1. State an outcome a user can observe

An agentic tester works the way a user does: it opens the app, clicks, types and reads what comes back. So a criterion has to end in something visible. A message, a new row in a table, a downloaded file, a changed total, an email in an inbox.

"Preferences are synced to the backend" gives it nothing to look at. "After switching the language to German and logging back in, the dashboard is still in German" does.

The same goes for visual requirements. Write what the user should see ("the error appears directly under the email field") and leave out references to a design file the tester can't open.

2. Name the start state and the role

Who is logged in, with which permissions, and where do they begin? Write "as a Viewer on a project someone else owns" where the ticket says "user", and the agent knows which account to log in with. A test case that runs as the wrong account fails for a dull reason: it can't see the part of the app that changed.

Permissions deserve the extra line. Who's allowed to do what, and what everyone else sees when they try, lives in people's heads and a policy file. A tester can't find it by clicking around.

3. Give it the data

Name the account, the coupon code, the file, the number of rows. If the data has to exist before the test starts, say so, and make sure it exists in the environment the agent runs against. If the agent should create it, say that.

Names count as data too. The ticket says "workspaces", the screen says "teams" and the code says "orgs". Add one line that maps them: workspaces = teams = orgs.

4. Say what's out of scope

Half-built features are normal in a pull request. The CSV export ships today, the PDF version next sprint, and the PDF button may already be on the page. Tell the agent which parts to leave alone, or it may report the unfinished one as broken.

5. Include the case that should fail

Happy paths are easy to write, so they're what gets written. Write the other one too: the expired code, the declined card, the role that shouldn't get in, the required field left empty. Say what the user should see in that case as well. A negative case with no expected result is as vague as "handle payment errors gracefully".

Where to stop: one criterion per outcome

There's a way to overdo all of this. Turn every line of every criterion into its own test and you end up with more tests than anyone can review. Spec tools can do the same thing to a ticket: writing on martinfowler.com, Birgitta Böckeler describes one turning a small bug into 4 "user stories" with a total of 16 acceptance criteria.

If two lines can only pass or fail together, they're one criterion. Leave the click-by-click steps out as well. The agent works out the route when it runs, and any steps you write today are steps you'll be correcting after the next redesign.

The checklist version, for a ticket template:

  • Ends in something a user can see
  • Names the role and the starting point
  • Names the data, and that data exists in the test environment
  • Has at least one case that should fail, with what the user sees
  • Lists what's out of scope
  • One line per outcome, no click-by-click steps
  • Sits where the tester reads it: the ticket description or the PR description

Acceptance criteria examples: before and after

Each row below is a ticket or a PR description, first as a teammate would write it and then as an agent needs it.

Where it livesWritten for a teammateRewritten for an agent
Jira story, description field"User can export invoices.""As a workspace admin on Billing → Invoices, with three paid invoices in October: Export CSV downloads a file, and the page confirms that three invoices were exported. A member without billing access doesn't see the Export button. Out of scope: PDF export."
Linear issue"Viewers shouldn't be able to edit projects.""Logged in as a Viewer on a project someone else owns: there's no Edit button. Opening the edit URL directly shows the no-access page, and the project name is unchanged afterwards. An Editor on the same project can still rename it."
PR description, bug fix"Fixes coupon bug.""What changed: the discount now recalculates when the quantity changes. What to test: cart with two items, apply SAVE10, raise the quantity to three. The discount line shows 10% of the new subtotal. The expired code OLD5 still shows 'This code has expired'. Out of scope: shipping and tax."
PR description, UI change"Improve onboarding UX.""What changed: a new account lands on a three-step setup checklist where the empty dashboard used to be. What to test: sign up with a fresh email and see the checklist; create a project and step 1 is ticked. An existing account with projects goes straight to the dashboard and never sees the checklist. Out of scope: the invite-teammates step (next PR)."

The invoice ticket gained a role, a start state with data in it and a negative case by role. The permissions ticket was already a negative case with no expected result, so its rewrite says what the Viewer sees. "Fixes coupon bug" assumed the reader knew what the bug was. "Improve onboarding UX" had no outcome in it at all, only a direction.

None of the rewrites says where to click. And the person who wrote each original already knew everything that's in the rewrite.

A PR description template you can copy

The headings below come from our PR testing best-practices docs: what changed, why, what to test, out of scope. The prompts under each heading are our suggestion. Nothing about the layout is mandatory.

## What changed
<User-facing behaviour in one or two sentences. No implementation detail.>

## Why
<User story, bug report or product goal. Ticket key: PROJ-123>

## What to test
- As <role>, starting from <page or state>, with <data>: <action>, then <what the user sees>
- Failure case: <condition>, then <what the user sees>
- Still works: <existing flow this change could break>

## Out of scope
- <What this PR deliberately doesn't change or finish>

Names (only if they differ): <ticket term> = <term on screen> = <term in code>

Filled in for the invoice export:

## What changed
Workspace admins can export the invoice list as a CSV from Billing → Invoices.

## Why
Finance teams re-type invoice totals into their own tools every month (BILL-482).

## What to test
- As a workspace admin with three paid invoices in October: Export CSV downloads
  a file, and the page confirms that three invoices were exported.
- As a member without billing access: the Export button isn't shown.
- A month with no invoices: the button is disabled and the page says there's
  nothing to export.
- Still works: opening a single invoice as a PDF.

## Out of scope
- PDF export of the whole list (next PR)
- Invoice numbering

If a coding agent writes your PR descriptions in CI, keep one rule. It can fill in "What changed" from the diff. "What to test" and "Out of scope" describe intent, and intent has to come from the ticket or from a person. Otherwise the code being checked is writing its own checklist.

How the agent reads your criteria in a PR review

In our product the review agent reads what's written, so intent nobody wrote down stays unknown to it. It won't compare a page against a design file. In Jira it reads the ticket's description and leaves custom fields and comments alone. The ticket has to sit in the Jira project selected in project settings, and only Jira Cloud is supported. Test data it creates stays behind afterwards, so cleanup is a test case or a rule you add.

Downloads need one more note. Our docs cover checking that a file was downloaded, so the invoice example stops there and confirms the count on the page. A criterion about what's inside the file, such as a three-row CSV whose total matches the page, is a check to try on your own app before you rely on it. Keep the file small if you do.

Within those limits, QA.tech, an AI testing platform, reads the acceptance criteria in the PR description and in the linked Linear or Jira ticket, and uses them to pick existing test cases and write new ones for the gaps. It runs them against the pull request's preview as part of PR testing and posts a verdict on the PR: pass, fail or unable to verify. Behind the verdict is the QA.tech run, linked from the PR, where the screenshots, logs and step trace are kept, with a recording if one was captured.

A few details decide whether your criteria get read at all. The ticket is found by its key, so put the key in the branch name (feature/BILL-482-invoice-export) or in a commit message, and connect the Jira integration or the Linear integration in project settings. For Jira, keep the criteria in the description field. If your tickets live in another tracker, put the criteria in the PR description, or pass them in the context field when CI starts the review.

The ticket and the PR description are read together. Criteria from the ticket are added to what the PR description says, so there's no need to paste them twice. The PR description carries the most weight and the linked ticket comes second, which puts the two of them at the top of the list in how an AI agent knows what to test on a pull request.

No format is required. The agent reads criteria as plain text, so bullets, prose and Given/When/Then are all read the same way. Each role your criteria name does need its own saved login. An expectation that holds for every change belongs in a PR Review Rule so nobody repeats it per ticket, and the rule library has a template for ticket acceptance criteria.

Acceptance criteria are also a plain way into the question "What is agentic QA?" In agentic testing an agent works from a goal and nobody scripts the steps. The criteria are those goals, written by the people who know what the feature is for. And the test cases an agent writes from them can outlive the pull request, which is how PR test cases become your regression suite.

Where Given/When/Then helps, and where it doesn't

Most teams know Given/When/Then from Gherkin, the language Cucumber scenarios are written in. In Cucumber's Gherkin reference, Given steps "describe the initial context of the system", When steps "describe an event, or an action" and Then steps "describe an expected outcome, or result".

Hold that against the five rules and you can see why it helps. Given forces a start state. Then forces an outcome. Someone who'd write "user can cancel" in prose will fill in a start state when the template has a slot for one, and a second scenario for the failure case is cheap to add. For a team whose criteria run thin, that discipline is worth having.

It stops helping in a few places.

  • The slots can be filled with nothing. "Given a user, When they export, Then it works" has the right shape and tells nobody anything.
  • It drifts into scripting. "When I click the blue Export button in the top right" is a step, and it breaks the day design moves the button. The same Cucumber page warns against it: "Try hard to come up with examples that don't make any assumptions about technology or user interface. Imagine it's 1922, when there were no computers."
  • Given can hide the data. "Given a customer with invoices" still doesn't say which customer.

So use it if your team already does, or needs the structure. Don't adopt it for the agent's sake. If your specs also drive a coding agent, remember that spec-driven development needs spec-driven verification: something other than the builder has to check the build against the spec.

When the ticket and the app disagree

QA teams looking at agentic testing ask a version of the same question. When the PRD and the implementation differ, which one wins? Which of the two does the tester treat as the source of truth?

Neither, on its own. The criteria say what was intended. The running app shows what was built. A test compares the two, and a failed test tells you they disagree. It can't tell you which one is wrong.

The app is wrong. The ticket says 10% and checkout applies 15%. That's a bug, and the fail is the process working.

The ticket is stale. The decision changed in a thread or a standup, the code followed and the ticket didn't. The fix is a one-line edit to the ticket or the PR description. It's worth making even though the code is right, because the next reader, whether that's a new hire or the agent testing the next PR, will trip on the same line.

The ticket is silent. Nobody wrote down what the discount should be. Now the only references left are the diff and the app as built, and a check against the app as built passes anything that works. A flow can complete without a single error and still do the wrong thing, and nothing goes red to tell you.

Somebody has to say which side is wrong, and that's whoever owns the ticket. In most product teams it's the PM. They don't have to write test steps. They do have to say how it should work, who's allowed to do what and what the customer was promised, because that's the stuff that doesn't live in the codebase. Or it lives there only as today's behaviour, and the code can't tell you whether anyone meant it.

One habit helps. When a check fails on a criterion and the app turns out to be right, fix the criterion before that PR merges. The reasoning behind checking a build against stated intent is in verification vs validation.

FAQ

How do you write acceptance criteria for an AI testing agent?

Write each one as an outcome a user can see, and give the agent what a new teammate would need: the role and starting point, the data to use, at least one case that should fail, and what's out of scope. Keep one criterion per outcome and leave the click-by-click steps out. Put them where the agent reads, which means the ticket description or the PR description.

Can AI generate an end-to-end test from a user story?

Yes, if the story carries enough to test against. An agent can take "as a workspace admin I can export last month's invoices as CSV", open the app and try it. What it can't supply is an expected result you never stated. A story with no acceptance criteria tends to produce a test that checks the flow completes, which is weaker than one that checks it did the right thing.

Can AI check a pull request against acceptance criteria?

Yes, though tools mean different things by it. Some AI code review tools compare the diff with the ticket text and flag requirements the code doesn't seem to address. A testing agent opens the running build and tries each criterion as a user would. Criteria about behaviour, meaning what the user sees after an action, need the second kind.

Should acceptance criteria use Given/When/Then for agentic testers?

Only if it helps your team write them. The format forces a start state and an expected outcome, which is where thin criteria are weakest. An agent doesn't need the format to read your criteria, and Given/When/Then filled in vaguely is still vague. Keep UI details like button colours and positions out of the When step.

How do you write test cases from acceptance criteria?

Take each criterion and add what it leaves out: the setup, the data and the expected result. One outcome becomes one test case, and a criterion with a failure case becomes two. With an agentic tester you can stop there, because the agent works out the steps when it runs. With a scripted framework you go on to write the steps and selectors yourself.

Where does the agent read acceptance criteria: the ticket or the PR description?

It depends on the tool, so check what yours fetches. A split that works with any of them: the ticket holds what the feature should do, and the PR description holds what this change did, what to test and what was left out. Watch for fields the tool doesn't read. Criteria kept in a custom field or a comment thread can be invisible to it.


Your team moves fast. Can your testing keep up?

Our QA agents test your product autonomously, so moving fast never means shipping broken. Start free or get a 30-minute demo.