# End-to-End Testing

Course: [Harness Engineering](https://www.devtutorials.ai/learn/harness-engineering) · Section: Long-Running Work
Source: https://www.devtutorials.ai/learn/harness-engineering/long-running/end-to-end-checks

> Why green unit tests and a list full of "passes": true can still hide a feature that is never turned on, and how an end-to-end check in the passes rule (Playwright for the React frontend, HTTP tests for the ASP.NET Core backend) proves each feature works in the running app.

## Green Tests, Broken App

Your harness now has a feature list, a handoff note and clean exits.

Each session builds one feature and runs the unit tests.

Then it flips `"passes"` to true.

At the end, every test is green. Every item in `feature_list.json` says `"passes": true`.

Now open the app, the way a user would. One feature is not there.

**End-to-end check** = a test that uses the whole running app, like a user does.

**e2e** = short for end-to-end. Commands like `pnpm e2e` run these checks.

For example:

- In the React Taskboard, it opens the page, types in the search box and reads the list.
- In the ASP.NET Core Pricing API, it calls `GET /orders?status=open` and reads the JSON.

### A Real-World Example

Think about building a new kitchen.

The electrician tests every socket on a bench. Each one works.

Then the kitchen is done. You plug in the kettle. Nothing happens.

One socket was never connected to the fuse box.

Agent work is the same:

1. **Unit tests** are the bench test. Each part works alone.
2. An **end-to-end check** is plugging in the kettle. It proves the parts are connected.

Click the button and open the **Sessions & App** tab.

- **Sessions:** all five end with green unit tests and a commit.
- **The App log:** one feature has code but is never turned on.
- **"(the list says it passes)":** the feature list is wrong about it.
- **The Terminal tab:** type `pnpm test`, then `pnpm e2e`. The first is green. The second names the broken feature. (For the .NET project: `dotnet test --filter Category=E2E`.)

**Simple lesson:** a feature can have code, a green test and a "passes" flag, and still not work.

## What You Will Learn

In this lesson, you will learn:

1. Why green unit tests can hide a broken app.
2. What wiring is, and why unit tests cannot see it.
3. The test pyramid: unit, integration and end-to-end tests.
4. How to write an end-to-end test for the React Taskboard (Playwright).
5. How to write an end-to-end test for the Pricing API (real HTTP calls).
6. How to make the end-to-end check part of "done".

**Lab vs real tools**

- The lab's agent is simulated, so it does the same thing every run. That is how your work can be checked.
- A real agent is less predictable. Try each idea on your own project too.

## Why Green Tests Can Lie

### Watch It Happen

Here is the Taskboard run from the lab, one step at a time.

Step through all five frames.

- **Frames 1 and 2:** every session is green. Every item says it passes.
- **Frame 3:** session 2 writes `search.ts`. It never adds the import line.
- **Frame 4:** `App.tsx` imports four features, not five.
- **Frame 5:** the running app. Search does not work.

### Unit Tests Check Parts, Not Wiring

**Unit test** = a test that calls one function on its own.

The search unit test calls the search function and checks the answer. It never starts the app.

**Wiring** = the line that turns a feature on in the app.

- React: `import "./features/search";` in `App.tsx`.
- .NET: `app.MapFilter();` in `Program.cs`.

The unit test does not look at that line. So it stays green when the line is missing.

### A Different Feature Each Time

The lab has three versions of the same job. In each one, the agent forgets a different line.

Step through all four frames.

- **Frames 1 and 2:** unit tests only. A different feature is unwired in each version.
- **Frames 3 and 4:** the end-to-end check is in "done". Every feature works, on both stacks.

**Simple lesson:** you cannot guess which line will be missing, so check the whole app every time.

## The Test Pyramid

### Three Kinds of Tests

- **Unit test** = one function or class on its own. It is fast. You have many.
- **Integration test** = two or more real parts together, like a service and a database. You have fewer. They are slower.
- **End-to-end test** = the whole running app, used like a user. You have the fewest. They are the slowest.

Only the top layer sees the wiring.

### Which Test Catches Which Bug

A bug inside one function shows up in its unit test.

A broken file stops the build. The unit tests go red too.

A missing import or a missing `app.Map<Id>();` is different. Every part still passes.

Only a test that starts the app sees it.

**Simple lesson:** wiring bugs only show up at the top of the pyramid.

## Test Like a User

### What an End-to-End Test Looks Like

You do not need to read every line. Each test does what a user would do, then checks the screen or the reply.

**Playwright** = a tool that drives a real browser. It opens the page, types, clicks and reads the screen.

Here is an end-to-end test for Search:

```ts
// e2e/search.spec.ts
import { test, expect } from "@playwright/test";

test("typing in search shows only matching tickets", async ({ page }) => {
  await page.goto("/");
  await page.getByRole("searchbox", { name: "Search tickets" }).fill("login");
  const rows = page.getByRole("listitem");
  await expect(rows).toHaveCount(1);
  await expect(rows.first()).toContainText("Fix login");
});
```

The test finds the search box by its **role** and **name**. A screen reader finds it the same way.

If Search is never imported, there is no search box. The test fails.

In `package.json`, `pnpm e2e` runs Playwright:

```json
"scripts": {
  "test": "vitest run",
  "e2e": "playwright test"
}
```

For an API, "like a user" means real HTTP calls.

It uses **xUnit** (the .NET test library). `WebApplicationFactory` starts the whole app in memory, with the real `Program.cs`. Then the test calls URLs.

```csharp
// Pricing.Tests/E2E/FilterE2ETests.cs
public class FilterE2ETests : IClassFixture<WebApplicationFactory<Program>>
{
    private readonly HttpClient _client;
    public FilterE2ETests(WebApplicationFactory<Program> factory) => _client = factory.CreateClient();

    [Fact, Trait("Category", "E2E")]
    public async Task Filter_returns_only_open_orders()
    {
        var orders = await _client.GetFromJsonAsync<Order[]>("/orders?status=open");
        Assert.NotEmpty(orders);
        Assert.All(orders, o => Assert.Equal("open", o.Status));
    }
}
```

The `Trait` is a label. `dotnet test --filter Category=E2E` runs only the tests with that label.

If `Program.cs` never calls `app.MapFilter();`, the filter is ignored. The test gets closed orders back, and it fails.

**Write failure messages for the agent.**

"No searchbox named Search tickets on the page" points straight at the missing wiring.

"Expected 1, got 0" does not.

**Simple lesson:** for an API, the end-to-end check is a real request to the real app.

## Make It Part of Done

### One Line in the Rules

A check helps only if the agent runs it before it says "done".

So put the end-to-end command in the line that flips `"passes"`.

For the React Taskboard:

```markdown
- Flip "passes" to true only after `pnpm test` and `pnpm e2e` both pass.
```

For the ASP.NET Core Pricing API:

```markdown
- Flip "passes" to true only after `dotnet test` and `dotnet test --filter Category=E2E` both pass.
```

Step through all four frames.

- **Frame 1:** the new rules line, marked in green.
- **Frame 2:** session 2 again. It adds the import, then runs `pnpm e2e`.
- **Frame 3:** the check runs once per feature, inside each session.
- **Frame 4:** both app logs side by side. Same green unit tests. Only one app works.

### Lab Exercise: Add the Check to Done

Change the rule that flips `"passes"`. Name the end-to-end command for your stack.

Click the button, change the rules file, then press **Check my harness**.

- **The check:** 5 sessions on each of 3 versions.
- **To pass:** every feature must work in the running app.
- **And:** no item may say `"passes": true` without proof.

**Simple lesson:** use the right command for the stack. `pnpm e2e` does nothing in a .NET project.

## The Seam Between Front and Back

### Two Green Sides Can Still Break

**Seam** = the place where the frontend and the backend meet.

Each side can pass its own end-to-end check. They can still break each other.

Say the API renames `status` to `state`. The Taskboard now shows empty labels.

Two checks guard the seam:

1. A **contract test** = the API response must match `openapi.json`.
2. A **full-stack end-to-end test** = Playwright opens the Taskboard while it talks to the real Pricing API.

### Where Each Check Runs

- **In each session, before "done":** the unit tests and the end-to-end check for that one feature. Keep these fast.
- **In CI on every change:** the full end-to-end suite, the contract tests and the full-stack tests.

The lab runs the per-session checks only. CI and contract tests happen in your own project.

CI (see **Tests and Linters**) runs the checks on every change.

**Never mock the thing you are trying to prove.**

To mock means to use a fake copy.

A Playwright test against a fake API cannot catch a renamed field.

A test that swaps out `Program.cs` cannot catch a missing `app.MapFilter();`.

**Simple lesson:** test against the real app, or the test proves nothing about it.

## TL;DR

1. **Unit test** = one part on its own. It cannot see wiring.
2. **Wiring** = the line that turns a feature on: an import in `App.tsx`, or `app.Map<Id>();` in `Program.cs`.
3. **End-to-end check** = the running app, used like a user.
4. **Test pyramid** = many unit tests, fewer integration tests, a few end-to-end checks.
5. **Frontend e2e** = Playwright, run with `pnpm e2e`.
6. **Backend e2e** = `WebApplicationFactory` and real HTTP calls, run with `dotnet test --filter Category=E2E`.
7. **Done** = flip `"passes"` only after the unit tests and the end-to-end check both pass.
8. **Seam check** = a contract test or a full-stack test, in CI.

## Common mistakes

| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| Unit tests only | "All tests pass, so it works." | Add an end-to-end check to done. | A feature has code but is never turned on. |
| E2E not in the passes rule | "E2E runs in CI anyway." | Flip passes only after e2e passes. | The list says a broken feature passes. |
| Wrong stack command | "`pnpm e2e` works everywhere." | Use `dotnet test --filter Category=E2E` for .NET. | The check never runs, so nothing is proved. |
| Mocked backend | "Mocks make it fast." | Use the real app for the check. | Renamed fields and missing routes slip through. |
| No E2E trait in .NET | "dotnet test runs everything." | Add `Trait("Category", "E2E")` and the filter. | The agent cannot run only the e2e checks. |
| CSS selectors | "Find it by `.search-input`." | Find it by role and name. | Tests break on every style change. |

## Check Your Understanding

## Exercises

These exercises practise end-to-end checks from this lesson.

Press **Check** when you are ready. Use **Show Hint** if you get stuck.

## What to Remember

- Unit tests check parts. They cannot see whether a feature is turned on.
- Wiring is one line: an import in `App.tsx`, or `app.Map<Id>();` in `Program.cs`.
- An end-to-end check uses the running app: clicks for the frontend, HTTP calls for the backend.
- Frontend: Playwright with `pnpm e2e`. Backend: `WebApplicationFactory` with `dotnet test --filter Category=E2E`.
- Flip `"passes"` only after the unit tests and the end-to-end check both pass.
- Check the seam with a contract test or a full-stack test in CI.

## Further Reading

- [Effective harnesses for long-running agents, by Anthropic](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents): testing features end to end with browser automation, as a user would.
- [Harness engineering: leveraging Codex in an agent-first world, by OpenAI](https://openai.com/index/harness-engineering/): letting the agent drive the running app to check its own work.
- [Harness engineering for coding agent users, on martinfowler.com](https://martinfowler.com/articles/harness-engineering.html): sensors at different levels of a test suite.

Next: **Evaluator Agents**. A second agent checks the work, so the builder does not grade itself.