# What Is a Test Harness? Components, Examples, and the Version Coding Agents Need

> A test harness lets tests run unattended and report: drivers, stubs, test data, an environment, a runner. Definition, test harness vs framework, examples.

Source: https://qa.tech/blog/what-is-a-test-harness · Published: 2026-11-16

---
A test harness is everything around the code under test that lets your tests run on their own and report a result. That means a driver that calls the code, stubs or fakes that stand in for the parts you aren't testing, test data, an environment to run in, a runner, and somewhere the results land. Your tests decide what gets checked; the harness gets them running and reporting without anyone babysitting the run.

## Test harness definition: drivers and stubs

The classic meaning is narrow. [Wikipedia's entry](https://en.wikipedia.org/wiki/Test_harness) opens with it: a test harness is "a collection of stubs and drivers" set up to help test an application or a component.

A **driver** calls the code under test the way the rest of the system would, passing in inputs and collecting outputs. A **stub** sits where a dependency would be and returns canned answers, so the test doesn't need the real database, payment provider or email service. Put the two together and you can test one module in isolation, repeatably, at any hour.

The same entry draws the line tightly: the harness simulates application functionality and "has no knowledge of test suites, test cases or test reports". In day-to-day use the word has grown to cover the whole setup around the tests:

- **Test data and fixtures** that put the system into a known state before each test.
- **Environment setup**, whether that's a container, a seeded database or a preview deployment.
- **A runner** that decides what runs, in what order and in parallel or not.
- **Reporting** that turns results into something people, or other tools, can act on: a CI check, a PR comment, a dashboard.
- **Teardown** that cleans up so the next run starts from the same place.

## Test harness vs test framework

People use the two words interchangeably, and they do overlap.

A **test framework** is how you write tests: JUnit, pytest, Jest, Playwright Test. It gives you the syntax for a test, assertions and hooks such as `beforeEach`. Each of these also ships a runner and reporters, which is why the words blur: the framework covers part of the harness.

A **test harness** is how tests get run against something real. It includes the framework, plus the data, environment, stubs, runner and reporting around it. Jenkins or GitHub Actions on its own only covers the part of a harness that starts the run.

A quick way to tell them apart: moving from Jest to Vitest barely touches your harness. Switch from mocked services to a real staging environment, and the harness changes a lot while the tests may not change at all.

## Test harness examples

**A unit-level harness.** You're testing a function that calculates shipping costs. The harness is a pytest file with a driver that calls the function with a table of weights and destinations, a stub that replaces the live carrier-rates API with fixed prices, and a CI job that runs it on every push and fails the build on a wrong number.

**An end-to-end harness for a web app.** You want to test checkout in the real UI. Now the harness has to deploy the app somewhere a browser can reach, create a test user through an API, put the payment provider into test mode, seed a product and a discount code, run the browser tests, collect screenshots and network logs when something fails, post the result on the pull request and delete the test user afterwards. In a setup like this, the work and the breakage tend to sit in those steps, not in the test code.

## The hard part is test data

When we ask teams on calls what makes their end-to-end tests painful, they talk about the data far more than the test code.

The same problem shows up in a few forms. Data gets used up: once a test has completed an order or verified an account, that record can't go through the flow again. Data is too interconnected to fake: one team we spoke to works with regulatory filings of two to three hundred related fields, where a mock that gets one relationship wrong proves very little. And parallel runs collide when several tests share one account and step on each other's orders.

There's no single fix, but a few rules help:

1. Create the data each test needs through an API at the start of the test, so no test depends on shared records.
2. Give every test its own user or tenant when tests run in parallel.
3. Keep a small set of long-lived accounts for read-only checks, and never let tests write to them.
4. Clean up after the run, or give each run a disposable environment.

## How to build a test harness

Roughly in this order:

1. **Decide what's under test and what isn't.** Everything outside the boundary gets a stub, a fake or a test-mode version.
2. **Make the starting state reproducible.** Fixtures, seeded data or an API that creates what each test needs.
3. **Pick where tests run.** For end-to-end tests, a preview environment per pull request is the safest choice, because each change gets its own URL and its own data. A shared staging environment works if tests don't fight over data.
4. **Choose a runner and wire it into CI**, with parallelism and retries set on purpose.
5. **Decide where results go.** A required check on the pull request does more than a nightly email nobody reads.
6. **Add teardown** before the leftovers pile up.

Then keep it boring: when a red check turns out to be a harness fault, fix the harness that day, before the team learns to rerun and move on.

## The harness coding agents need

In the last year the word has picked up a second meaning. Coding agents run inside a harness too: the instructions, tools, linters, tests and checks around a model that turn it into something that ships code. Birgitta Böckeler's [harness engineering article](https://martinfowler.com/articles/harness-engineering.html) on martinfowler.com splits it into guides, which steer the agent before it acts, and sensors, which observe afterwards so it can correct itself.

Most of the sensors she lists work on the code itself: linters, type checkers, structural tests, AI code review. On whether the application "functionally behaves the way we need it to", she calls that "the elephant in the room", and says leaning on AI-generated tests is "not good enough yet". We've argued the same in [the sensor an agent harness is missing](https://qa.tech/blog/harness-engineering-behaviour-sensor): an agent's own tests come from the same reading of the ticket as its code, so they share its blind spots.

That gap is a test harness problem in the old sense. Something has to deploy the change, put the app into a known state, exercise it the way a user would and report back in a form the agent can act on. Our CEO and co-founder [Daniel Mauno Pettersson](https://www.linkedin.com/in/danielmaunopettersson/) describes it as a counterparty to the agents producing your code: something outside them that owns quality. He frames the build-or-buy question as one of core business. A QA harness is a product in its own right, and it probably isn't the one you sell.

## Where an external testing agent fits

You can build that behaviour layer yourself. The alternative is to plug in a testing agent that already has the runner, the reporting and the browser, and point it at your preview environments.

[QA.tech](https://qa.tech/product/pr-testing), an AI testing platform, is one of those, and it leaves parts of the harness to you. Its web testing runs on Chromium and nothing else. It has no access to your database, so it can't push test data in directly; teams give it an API or a sign-up flow to create test accounts. It doesn't automatically clean up the test data it creates, so plan for teardown: a seed script that resets the environment, or a cleanup step in the test that deletes what it made through the UI or an API call. And it needs somewhere to run: a preview URL per pull request, or a staging environment after merge.

Within those limits, it reads the change, picks existing tests for the user-facing parts, writes new ones where there are gaps, runs them against the preview deployment and posts a pass, fail or unable-to-verify verdict on the PR. A coding agent can read that result over MCP. That covers the driver, runner and reporting parts of the harness; the data, environment and teardown are still yours.

## FAQ

**What does test harness mean?**
Everything that lets tests run automatically and report a result: something to call the code, stand-ins for the parts you aren't testing, test data, an environment and a runner. The tests themselves only decide what gets checked.

**What is the difference between a test harness and a test framework?**
A test framework, such as JUnit, pytest or Playwright Test, is how you write tests. A test harness includes the framework plus the test data, stubs, environment, runner and reporting that let those tests run against your system.

**What is the difference between a driver and a stub?**
A driver calls the code under test, standing in for whatever would normally call it. A stub is called by the code under test, standing in for a dependency such as a database or an external API and returning fixed answers.

**Is a CI pipeline a test harness?**
Not on its own. CI is the piece that starts the run and collects the result. The harness also includes the test data, stubs, environment and reporting that make the run meaningful.

**What is an agent harness?**
It's the setup around an AI coding agent, including its instructions, tools, linters, tests and review steps, that turns a model into something that can ship code. A test harness is one part of it: the part that checks whether the change actually works.

**Do you need a test harness for end-to-end tests?**
Yes. End-to-end tests need a deployed app, a known starting state, test data and somewhere to report results. That setup is the harness, and for end-to-end testing, building it can take more work than writing the tests.
