Build your own QA agents when the app is small, one team owns it, and the checks run at a developer's desk while they code. Buy when the same checks have to run unattended on every pull request, across teams, with logins, test data, parallel browsers and evidence that someone other than the author can trust.
Should you build QA agents with a coding agent and Playwright, or buy a platform?
If you're asking this, someone on your team has probably built something already. A developer pointed a coding agent at a browser tool, asked it to test the signup flow, and it did. Your team doesn't need convincing that it can be done.
The buyers who frame this best ask two things. Could we build it? Probably yes, given what a coding agent can produce in a week. Do we want one more thing to maintain? Getting an agent to click through your app takes an afternoon on either route. What you're deciding is who builds and maintains everything around it.
Several teams we've spoken to this autumn asked themselves the same thing and went off to build it, and some of them were right to. Nothing below depends on your stack: the Claude-specific comparison is Claude plus Playwright versus a platform, and the question before this one, whether the check should sit outside the coding agent at all, is Can You Run a Software Factory Without a Verification Layer?.
What "building it" means in 2026
The kit is easy to get, and where teams differ is in how much of the right-hand column they've thought about.
| Part | Typical choice | What you own afterwards |
|---|---|---|
| Coding agent | Claude Code, Cursor, Codex | Model upgrades, and re-testing your prompts after each one |
| Browser tool | The Playwright MCP server, a Playwright CLI, or the agent's own browser integration | Version bumps, permissions, browsers in CI |
| Skills and prompts | Instruction files: how to log in, where things live | Keeping them true as the app changes |
| CI wiring | A job that starts the agent on a pull request and turns its output into pass or fail | Secrets, timeouts, budgets, the result format |
For the browser tool we have a guide to what Playwright MCP is and how to set it up, though the kit keeps moving. As of 16 Nov 2026, Playwright's docs list three test agents that ship with it, a planner, a generator and a healer. Microsoft's README for the MCP server now says coding agents increasingly favour CLI workflows exposed as skills, because they use fewer tokens.
thoughtbot published a QA proof of concept built on Playwright MCP in January 2026 and wrote that the exercise had proved the concept and had potential with further exploration. Nobody should tell you this can't be built.
Where DIY is good enough
Building is the right call in more situations than a vendor's blog tends to list. DIY is probably enough if:
- the app has a handful of flows and one team owns all of them
- the checks run locally, while a developer is coding
- the person who reads the result is the person who made the change
- you have platform engineers with time set aside for the harness
- the tests have to live in your repository as code
- what you want is a few dozen fixed checks that run the same way every time (a script does that faster and with no variation, and a coding agent can generate it)
- nothing is allowed to run outside your network
Anthropic's Claude Code docs list web app testing among the uses of the Chrome integration, in the developer's own browser with their own login. That local loop is the strongest case for DIY, so keep it whatever else you decide. The best-practices page in the same docs explains why it matters. Without something to run, "looks done" is all the agent has to go on.
Ownership is a real reason as well. Engineers on small teams say they could set up something similar with their own agents, and that they want the tests in their own code. They're right that they could. On ownership, the most a platform like ours hands back is the test case definitions, exported as JSON. It can't give you a suite that runs from your repository.
Some parts can be easier to build than to buy. Code review was one for a team whose self-built merge request reviewer beat the commercial tools they trialled it against. What they still lacked was the part where something logs in and uses the product.
What teams underestimate
None of what follows comes with the agent. You build it or you go without.
Agents that loop, get lost and burn tokens
An agent with no map of your app guesses how to reach the billing page. Sometimes it guesses wrong and keeps trying. A practitioner writing on DEV built on Playwright's test agents with Claude Code, and the planning step used up most of a subscription window before the generator had written a single test. The thoughtbot proof of concept hit context window limits more than once. So the harness needs repeat detection and a step budget with a hard stop, and somebody has to write them. Our Playwright MCP token cost breakdown shows where the tokens go when they're missing, and The Full AI Cost of Shipping a Feature sets that spend beside the rest of what a feature costs.
A world model that drifts
The fix for getting lost is a map: which pages exist, how you reach them, what an admin sees that a viewer doesn't. Teams write this into a skills file, or have the agent crawl the app and save notes. It works until the UI moves. Then the agent quietly goes back to guessing, and no test reports it. The tools drift too. Playwright's docs say its agent definitions should be regenerated whenever Playwright is updated. A maintained picture of the app is one of the inputs in how an agent knows what to test on a pull request, and somebody has to keep it current.
Logins, 2FA, email and test data
At a desk, auth is free. Per Anthropic's docs, Claude Code's Chrome integration shares your browser's login state and pauses to ask you when it meets a login page or a CAPTCHA.
In CI there's no you. The agent needs a test account for each role. It needs a way to produce a one-time code, an inbox it can read for the verification email, and data that's already there when the run starts. Then it needs all of that per environment, with the secrets kept out of the transcript. Each piece is small. There are a lot of pieces.
Parallel runs
One agent driving one browser is fine on a laptop. A pull request check that covers several flows needs them running side by side, each in its own session, or the tests log each other out and you spend a morning chasing a failure that was never a bug. That means browser infrastructure with a concurrency cap, so the agents don't flatten staging.
Evidence and triage
A red result from an agent is a claim. Somebody has to decide whether it's a bug, a flake, a stale test or the agent getting confused, and an exit code won't tell them. In the same DEV write-up, the 2 bugs the agents flagged were not real. Playwright's docs say the healer agent can end on a skipped test "if the healer believes that functionality is broken", which is a judgement a person then has to review.
So you build the evidence. That's a screenshot per step, the console and network logs, expected against actual, and somewhere to keep it all. Then you build the second look, a rerun or a second agent that checks a failure before it reaches a human. Skip that and the red results stop getting opened.
Nobody validates what the home-built agent keeps
This one looks like success. An agent that writes test cases from every merged pull request will do exactly that, and a home-built one can grow a suite faster than anyone reads it. At that point nobody can say whether the new test cases check anything real.
There's a structural version of the same problem. When the agent that wrote the code also writes the tests, those tests are evidence for the agent. They tell it whether its own assumptions held. Whether the change meets your bar is a separate matter, and its author is the wrong one to rule on it.
"Another type of maintenance"
Generating tests was never the expensive part. Daniel Mauno Pettersson, our co-founder and CEO, brought up the other route himself on a call, the one where a coding agent writes your test scripts. "And that is also doable," he said, "it's just another type of maintenance needed."
Swap the scripts for an agent loop and the maintenance moves again. You stop fixing selectors and start maintaining prompts, the map, the accounts, the budget caps and the triage step, and you re-test all of it when the model underneath changes. Someone owns that by name, or it rots.
Build or buy: a decision table
Answer each row for the check you want on pull requests, since that's the one that has to work unattended.
| Question | Build if | Buy if |
|---|---|---|
| Where does the check run? | At a developer's desk, or on one team's pull requests | On every pull request, across several teams |
| Who owns the harness? | A named platform engineer with time set aside | It would be someone's side project |
| What does login involve? | One test account, no second factor | Several roles, 2FA, emailed codes |
| How many flows per change? | A few, one after another | Many in parallel, inside the review window |
| Who reads a failure? | The author, straight away | A reviewer or another agent, who needs evidence |
| Who checks what the agent adds to the suite? | Someone reads every new test | Nobody has time, so you need a filter and labels |
| Must the checker be independent of the coding agent? | Self-checks are enough for now, or you'll keep a separate checker the coding agent can't edit | Yes, and you'd rather not police that separation yourself |
| Must the tests live in your repo as code? | Yes | No. Plain-language test cases in a platform are fine, as long as the definitions can be exported |
| Can anything run outside your network? | No | Yes, a hosted service can reach staging |
If your answers sit in the middle column, build, and don't let a sales call talk you out of it. If they sit on the right, you'd be building a product. Mixed answers point to doing both. The developer's local loop and the fixed critical-path checks you run on every change stay in-house, and the layer that works out what to test from each pull request is bought. If the table has to go to a budget holder, the CTO's side of the QA bottleneck is the short form of the buy case.
What a platform adds, and what it won't do
Start with the limits, using the platform we build. It won't put tests in your repository. Test cases live in the platform in plain language. You can take the definitions out: name, goal, expected result and steps export as JSON through the API or the CLI, and the docs show how to turn that into CSV. What you get describes each test case. It isn't Playwright code, and existing scripts can't be imported. If owning the suite as runnable code is a requirement, that row of the table decides it.
Runs happen in its cloud. Most CAPTCHA types stop it, as they stop any agent with nobody there to solve them. And its selection isn't fixed: ask for the same review twice and the list of test cases may differ a little, so for checks that must run identically every time, scripts are the better tool.
QA.tech, an AI testing platform, provides the layer around the agent as a hosted service. On a pull request its agents work out what the change touches, run the saved test cases that apply plus new ones for anything uncovered, in cloud browsers against the preview, and report pass, fail or unable to verify. Here's what that takes off the list above, as documented on 16 Nov 2026.
| What you'd otherwise build | What the platform does |
|---|---|
| A map of the app | A crawl builds a graph of pages and interactions, used when test cases are created and pull requests are reviewed |
| Something that carries over between runs | The test cases themselves, plus the rules your team writes |
| Logins, 2FA and email | Saved configs hold an account per role; the agent generates a fresh 2FA code at login and has an inbox for verification emails |
| Parallel browsers | Independent test cases run side by side, each in its own isolated browser session |
| Evidence | A step trace with screenshots, console and network logs, and a recording when captured, kept in the run and linked from the PR |
| A second look at failures | On test plan runs, a review agent reruns suspected flakes and separates bugs from outdated test cases |
| A filter on what joins the suite | After merge, an agent working from an editable prompt promotes test cases worth keeping, labelled regression and auto-added |
| A merge gate | A required check, on GitHub only |
Config values are passed to the model during a run, so use dedicated test accounts with limited permissions. The post-merge agent is on by default for new GitHub integrations, has to be switched on for existing ones and on Azure DevOps, and isn't available on GitLab. Some merges add nothing to the suite, and what it does promote still needs a human look, the weekly pass described in How PR Test Cases Become Your Regression Suite.
None of this replaces the local loop. The coding agent you already use can start a run and read the results over MCP, which keeps the checker outside the agent that wrote the code. The aim is confidence to merge at the pace your coding agents open pull requests.
If you've already decided to buy and the open question is how to choose an end-to-end testing platform, the buyer's guide to evaluating agentic testing tools picks up from there. Otherwise, run the pilot below.
What to pilot for two weeks, either way
Don't decide from a demo, ours or your own. Run the same two-week pilot on the route you're leaning towards, or on both. Scope it identically:
- One application, and every pull request it gets in the two weeks.
- One flow behind a login that needs a second factor or an emailed code.
- A handful of bugs you reintroduce on purpose, without telling whoever built the setup.
- One UI change in week two that moves or renames something on a covered flow.
- For the build route: one engineer, a fixed number of days, and a log of every hour spent on the harness.
Then measure the same things on both sides.
| Measure | What it tells you |
|---|---|
| Hours spent on the harness (build) or on setup and tuning (buy) | The upkeep line. Watch whether it falls in week two |
| Share of pull requests that got a result before a human reviewed them | A check that arrives after review changes nothing |
| Seeded bugs caught | The only ground truth you'll have |
| False alarms and runaway runs, with minutes lost to each | Whether people will keep reading the check |
| Failures someone other than the author could judge from the evidence alone | Whether the result travels |
| What broke after the week-two UI change, and who fixed it | The maintenance you're signing up for |
| Spend per pull request, read off your own bill | The only cost figure that's yours |
Read the build results with one question in mind. Are the harness hours still climbing in week two? If they are, you've met the maintenance early. If they've flattened and the seeded bugs were caught, you have your answer, and it cost you two weeks.
FAQ
Should we build our own end-to-end test automation with Playwright and a coding agent, or buy a testing platform?
Build if one team owns a small app, the checks run while a developer is coding, and a named engineer has time to own the harness. Buy if the check must run unattended on every pull request for several teams, with real logins, parallel browsers and evidence a reviewer can trust. Doing both is a sensible outcome.
When is a DIY QA agent good enough?
When the person who reads the result is the person who wrote the change. A developer's agent checking its own work in a local browser needs no shared logins and no audit trail. Once someone else has to rely on the result without having watched the run, you need evidence and a second look at failures, whether you build them or buy them.
What can Playwright MCP not do for test authoring?
It gives an agent a browser. It doesn't give the agent a map of your app, test accounts, a place to keep evidence, a way to run many sessions at once, or anyone to check the tests it writes. Those are yours to build, on top of the setup and limits in Playwright MCP explained.
Is Playwright MCP free?
Yes. The server is open source under the Apache 2.0 licence, so there's no licence fee. You pay for the model calls it drives and for the engineering time around it.
Which test tools work with Claude Code and Cursor?
There are two kinds. Browser tools the agent drives itself, such as Playwright through MCP or a CLI, and cloud testing platforms that expose an MCP server so the agent can start a run and read the results. The first kind keeps everything local. The second keeps the check independent of the agent that wrote the code. Both kinds are on our list of QA tools for coding agents.
What does it cost to maintain a Playwright test suite in-house versus using an AI testing tool?
The in-house bill has lines people forget: engineer hours on the harness, model tokens per run, browser infrastructure, and the time spent deciding whether each failure is real. A platform folds the first three into its price. Whether it shortens the fourth is what a pilot should show. You're left with the setup and the rules. The cost lines, with the public figures that exist for them, are in The Full AI Cost of Shipping a Feature.
Can the agent that writes the code also test it?
It should test its own work as it goes. That's how it finds out whether its assumptions held. Deciding whether a change can merge is a separate job, and it belongs to something the coding agent doesn't control, for the same reason people don't approve their own pull requests.
