# The Tokenburner: how teams burn budgets testing with Claude and Playwright without a harness

> Where the tokens go when a coding agent drives a browser, why cheaper models don't fix it, and the suite-scale arithmetic a harness amortizes.

Source: https://qa.tech/blog/tokenburner · Published: 2026-10-15

---
In March 2026, Anthropic acknowledged that Claude Code users were hitting usage limits [faster than expected](https://www.theregister.com/2026/03/31/anthropic_claude_code_limits/). One developer's [GitHub report](https://github.com/anthropics/claude-code/issues/41930) put a five-hour session at 19 minutes. A [Hacker News commenter](https://news.ycombinator.com/item?id=48883796) described the fast version: "it launched 102 agents and blew my $120 quota in 6 minutes."

Part of that spike traced to Claude Code itself: the same GitHub thread reported two prompt-caching bugs inflating token use by 10 to 20 times, and Anthropic had trimmed session limits at peak hours around the same time. The headline incident was partly a bug. The structural point survives without it. A bug drained one engineer's quota in minutes while he watched. A DIY agentic test suite does the same thing on purpose, unattended, hundreds of times a day. A developer burning a quota is sitting there watching it. A CI job is not.

<div class="qa-highlight"><strong>Takeaways</strong><ul><li>A browser agent's context grows every step: each page snapshot stays in the conversation and is re-read on every following turn.</li><li>Single-author traces put a Playwright MCP test near 100,000 or more tokens, against roughly 27,000 for a CLI-based loop. Anecdotal, not a controlled study.</li><li>In one published benchmark, harness choice moved tokens per solved task by up to 70x on the same model.</li><li>Inference cost per unit of capability falls about 10x a year, and agent workloads inflate consumption faster than that.</li><li>At 50 to 500 runs a day, none of the above is a rounding error.</li></ul></div>


## The token cost of Playwright MCP and Claude

Where do the tokens go? Mostly into re-reading.

Playwright MCP drives a browser for an LLM by handing back the page as a structured accessibility tree, one `browser_snapshot` at a time. Playwright's own docs put a snapshot at a few hundred tokens for a simple page. Real applications are not simple pages. On a dense dashboard the tree balloons, and every snapshot you take stays resident in the conversation history. Step two re-sends step one's snapshot. Step twelve re-sends all eleven before it. Nothing is de-duplicated.

The clearest write-up comes from engineer [Kuljeet Singh Shekhawat](https://medium.com/@7003425114klp/one-screenshot-232-000-tokens-0b37783438c7), who wired Playwright MCP into Claude Code and ran a 15-test sheet. Six tests in, 75% of a 200,000-token context window was gone. In the trace, a single `browser_take_screenshot` call had consumed 232,000 tokens by itself. His summary of the mechanism: the model is rereading every snapshot, on every turn. [Prrammod Dutta's benchmark](https://scrolltest.medium.com/playwright-mcp-burns-114k-tokens-per-test-the-new-cli-uses-27k-heres-when-to-use-each-65dabeaac7a0) landed in the same place: by step twelve the conversation carried 90,000 tokens of stale snapshots from screens the agent had already left.

Three cost centers, then:

| Cost center | What it is | Why it grows |
|---|---|---|
| Snapshot accumulation | Accessibility trees kept in history | Re-read on every subsequent turn (the slow burn) |
| Screenshots | Image payloads for visual steps | Large and per-step; one call can dwarf a context window (the fast burn) |
| Startup tax | Tool definitions and system prompt loaded at session start | Re-sent as part of the conversation floor every single turn (see the harness numbers below) |

The point is structural: the serving system has to replay the conversation to keep it coherent, and replay is metered. For the mechanics of the MCP itself, see [what Playwright MCP is and where it stops being a test suite](https://qa.tech/blog/getting-started-with-playwright-mcp).

![Line chart over 12 steps: a naive loop's tokens per turn climb as snapshots accumulate; a harness stays flat at the floor with one spike on a real UI change](https://qa.tech/blog-assets/tokenburner-figure-1-context-growth.png)

*Figure 1. In a naive loop, every step re-carries every earlier snapshot. The floor you re-send each turn is what sets the bill. Illustrative numbers.*

## Why cheaper tokens are not a rescue

Inference cost for a fixed level of capability really is falling fast. Andreessen Horowitz's ["LLMflation" analysis](https://a16z.com/llmflation-llm-inference-cost/) puts the drop at roughly 10x a year. The catch is that nobody holds capability fixed. Teams move to the newest frontier model, and frontier is never the cheap one.

As investor Ethan Ding argues in [AI Subscriptions Get Short Squeezed](https://ethanding.substack.com/p/ai-subscriptions-get-short-squeezed), the same task that once ran as a single completion now runs a whole agent loop, so consumption climbs even as per-token prices drop; in his words, the math behind flat-rate pricing has broken. From the practitioner side, Sourcegraph's Steve Yegge has pegged [heavy agent use at roughly $10 to $12 an hour](https://sourcegraph.com/blog/revenge-of-the-junior-developer), which for a working developer lands around $100 to $300 a day. Now stop thinking about a workday and start thinking about a CI schedule that never clocks out.

Budgeting on falling token prices assumes constant work per token. Agent testing grows the work per run, unattended.

## The suite-scale arithmetic

The arithmetic, so you can substitute your own numbers. Illustrative; the methodology note below lists the assumptions.

The burn for one full run of a DIY loop is roughly:

**tokens per step × steps per flow × flows per suite × price per token**

Then multiply by runs per day. Take a modest suite: a blended 8,000 tokens per step (small snapshots plus the growing floor they ride on), 6 steps per flow, 30 flows, at an input price around $3 per million tokens.

| Scenario | Per run | At 20 runs/day | At 200 runs/day |
|---|---|---|---|
| DIY loop, re-reasons everything | ~1.44M tokens (~$4.30) | ~$86/day | ~$864/day |
| Harness: reason once, replay the rest | reasoning only on changed paths | small, flat base | small base plus spikes on real UI change |

*Methodology: 8,000 tokens per step is a blend of a modest accessibility-tree snapshot and the accumulated floor re-sent each turn; 6 steps per flow and 30 flows describe a small suite; $3 per million is a mid-tier input price in October 2026. Output tokens, retries and screenshots are excluded, so the figure is a floor, not an estimate of your bill.*

The biggest lever is the scope of what runs, and it is not in the table. A documentation-only change should cost close to nothing. In a naive loop it costs the same as a change to checkout, because nothing classified it first. Risk-weighted scoping cuts more from the bill than any per-token optimization you can do.

## What a harness amortizes

The model is the cheap part of a harness. The clearest proof comes from an independent benchmark by engineer Eishan Lawrence, [surfaced by The New Stack](https://thenewstack.io/agent-harness-token-costs/): across 12 harness configurations running an identical model, tokens per solved task ranged from about 3,500 (Aider in architect mode) to about 292,000 (OpenClaw). The suite ran on two unrelated models and the ordering of harnesses barely shifted between them, which is what points at the harness rather than the model. That benchmark measures coding harnesses, not browser-testing harnesses; the mechanism generalizes, the exact numbers are borrowed.

The startup tax makes the pattern concrete. The same benchmark reported roughly 700 tokens for Aider in architect mode against roughly 26,000 for OpenClaw, before any real work happens, resent on every turn. A 40x floor paid once would be tolerable. Paid per turn, it is not. That is the "startup tax" row from the cost table above, priced.

Why does a naive loop re-reason at all? Because an agent keeps nothing between runs unless something around it does. As QA.tech co-founder and Chief AI Officer Vilhelm von Ehrenheim (fifteen years in machine learning, previously at Klarna) put it in a September talk to the CTO Craft community: an agent forgets everything you told it unless someone has built the memory and context handling around it. That is the one-line reason the same agent tours your application from scratch on run 200 exactly as it did on run 1.

Stopping that waste is what a harness does. It models the application once (a crawl builds a persistent map that later runs reuse, instead of rediscovering the UI from scratch), routes complex flows to a heavier agent tier and everything else to a faster one, and confirms results with deterministic checks (network assertions, email and file-download checks, API calls) rather than more token-hungry reasoning. It can also keep the loop inside the tools you already use: through an [MCP server](https://qa.tech/product/mcp), a coding agent like Claude Code starts a run and reads results instead of burning its own context to drive the browser. [Past the Bottleneck](https://qa.tech/ebooks/past-the-bottleneck) puts the alternative in one line: an agent that explores from zero every run is slow, inconsistent, and expensive. An agent that starts from zero pays for the guided tour every single time.

That changes the shape of the bill, which is easiest to see by pricing model:

| Model | You pay by | Punishes | Best for |
|---|---|---|---|
| Per token (DIY Claude + Playwright, [compared here](https://qa.tech/compare/qa-tech-vs-claude-playwright)) | Tokens consumed per run | Suite size × run frequency | A few flows, one owner, no CI schedule |
| Per runner-minute (some AI platforms) | Execution wall-clock | Long or parallel runs | Teams that already tune for speed |
| Per test-case execution ([QA.tech](https://qa.tech/pricing) and some others) | Tests actually run | Running junk you did not need | Predictable CI budgets at scale |

Per-execution pricing has its own exposure: an entry price is not your cost at a real workload, so ask any vendor (ours included) what a 150-test suite running 100 times a week actually consumes. And if you have a handful of flows, one owner, and no CI schedule, the burn is negligible: build it yourself with Claude plus Playwright, which is fast at scaffolding tests. The curve only bites at suite scale.

## What actually caps the bill

Runaway spend is not unique to a DIY loop. Any agentic setup can do it, including a managed one, and anyone who has watched a bill spike knows it. The question is whether the controls exist and whether you had to build them yourself.

| Control | What it does | What you build yourself without it |
|---|---|---|
| Scope by risk | Classifies the change first, so a docs-only pull request triggers almost nothing and a billing change triggers everything | A change classifier and a policy for what each risk level runs |
| Choose the trigger | Label-based or on-demand runs, so not every push spends automatically | Trigger routing in CI, plus a clean way to fall back to on-demand while you tune |
| Cap concurrency | Per-environment limits on parallel runs, which protect both the bill and the staging server | Queueing and rate limits around your own test infrastructure |
| Reuse, do not rediscover | A persistent model of the app and cached decisions on unchanged pages | Storage, cache invalidation, and a crawl you keep in sync yourself |
| Actionable evidence | Failures carry expected versus actual, screenshots, the stack trace, and the API calls that failed, so the agent retries less | A verdict format and all the plumbing to capture it |

Two of those deserve a note. Risk scoring is the biggest lever by far: score the change, then decide what to run, and some teams now auto-merge low-risk pull requests on that score because human attention is worth more elsewhere. Actionable evidence is a cost item: a green-or-red result forces manual digging and blind retries, and retries are a line in the cost model above. A verdict an agent can act on directly means fewer full re-runs.

Concurrency is also where the DIY loop hits a wall on your own machine. You cannot run ten browsers on the laptop you are trying to work on. Von Ehrenheim's own prediction is that local development fades for exactly this reason: one task in front of one model made local sense, many agents in parallel do not, because you end up bottlenecked serving your own application from your own laptop.

## You do not need a big regression suite

The failure mode we watch teams walk into is volume too early: hundreds of unstable tests re-running everything, everywhere, on every push.

A calmer shape works better and costs less. Keep a small scheduled [regression suite](https://qa.tech/blog/what-is-regression-testing) of the flows that must never break, and run dynamic testing on every pull request where the diff decides what actually runs. Chasing 100% coverage works against you here: more surface to re-run for less signal.

Not everything needs a model, either. As von Ehrenheim puts it: don't bring a huge hammer to a small nail. Some checks are invariants, things that should never change and can be codified cheaply: an architectural rule about which modules may import which, a contract that blocks a merge on a breaking API change, a constraint that keeps personal data out of the logs. A linter is the simplest invariant. They run on every change for almost nothing. Judges, a model reading the result, are for what is too fluid to codify. Run a model over everything and you pay frontier prices for work a codified check does for free.

## If you would rather not build it

Maintaining the loop, the world model, and the token budget yourself is real work, and it drifts out of alignment every time the UI moves. That is what a managed agentic testing platform like QA.tech is for: it does the amortization as a product and ships the controls above. It will not make a runaway impossible, nothing does; it makes the caps someone else's code. If your CI is already competing with your engineers for tokens, [a short demo](https://qa.tech/demo) is the fastest way to see whether the shape fits.

The same loop that drained one engineer's quota in minutes is what a DIY suite runs hundreds of times a day, and a nightly job notices at the end of the month. Your CI should not be competing with your engineers for tokens.

## FAQ

**How many tokens does an AI browser test use?**
There is no fixed number, but anecdotal single-author traces put a Playwright MCP loop near 100,000 or more tokens per test, driven mostly by accumulated page snapshots.

**Why does Playwright MCP use so many tokens?**
Because each page snapshot stays in the conversation and is re-read on every following turn, so context and cost grow with every step you take.

**Is per-token or per-execution pricing better for testing?**
Per-execution pricing makes a CI budget predictable because cost tracks tests run rather than tokens consumed, though you should still ask what a real workload actually costs.

**Do cheaper models fix agent testing costs?**
No, because agent workloads inflate token consumption faster than per-token prices fall, and teams keep moving to the newest frontier model.

**How do you keep agentic testing costs down?**
Scope runs by risk, cap concurrency, reuse a persistent model of the app, and reserve the model for what cannot be checked with a cheap deterministic assertion.