# Continuous Improvement

Course: [Harness Engineering](https://www.devtutorials.ai/learn/harness-engineering) · Section: Scale and Maintain
Source: https://www.devtutorials.ai/learn/harness-engineering/scale-maintain/improvement-loop

> How to turn every agent mistake into one new check in the right layer, and how ablation (removing one part at a time and re-running every check) shows which parts of your harness carry weight and which only cost context.

## Keep the Harness Small and Useful

Over the course, your harness grew.

It has a test command, a guard for the production file, a skill and a lint message. It also has a hook, a progress file and more.

Some lines were added in a hurry.

Some were copied. Some were never needed.

**Improvement loop** = turn each agent mistake into one new check, then rerun everything.

**Ablation** = take one part out, rerun every check, and see what breaks.

Every line costs context. A line that does nothing still takes up room.

For example, the lab's harness has 9 parts:

- In the React Taskboard: a `pnpm test` line, a guard for `.env.production`, a skill and an ESLint message.
- In the ASP.NET Core Pricing API: a `dotnet test` line, a guard for `appsettings.Production.json`, a skill and a `BannedSymbols.txt` message.
- In both: a block-skip hook, `progress.md`, a vague line, a read-first line and a copy of the rules.

### A Real-World Example

Think about a backpack for a long hike.

After every trip, you add something.

- A rain jacket, because you got wet.
- A second torch, "just in case".
- A copy of the map you already carry.

After a year, the bag is heavy.

So you put everything on the floor. You ask about each item: "Did I need this on my last trips?"

The rain jacket, yes.

The second torch and the map copy, never.

You put back only what you needed. The bag is lighter, and nothing that matters is missing.

An agent harness is that backpack. Ablation is putting everything on the floor.

Click the button and open the **Sessions & App** tab.

- **Each row:** one part was removed. Then every check ran again.
- **Load-bearing:** a check broke without that part.
- **Can go:** every check still passed without it.
- **Cost:** each row also shows what the part costs: tokens for the agent to read, and extra steps per build. The progress note costs steps, but it saves whole sessions of redone work.

**Simple lesson:** if nothing breaks without a part, that part was not doing anything.

## What You Will Learn

In this lesson, you will learn:

1. How to write an incident down.
2. How to tell a guide gap from a sensor gap.
3. How to pick the layer for a fix: rules, skill, test or lint, or hook.
4. How to run an ablation and read its result.
5. How to keep the harness lean without losing a single check.

**Lab vs real tools**

- The lab's agent is simulated, so it does the same thing every run. That is how your work can be checked.
- A real agent is less predictable. Try each idea on your own project too.

**Simple lesson:** the lab measures each part, so you do not have to guess.

## The Improvement Loop

The loop has six steps:

1. **Incident:** write what the agent did, and what it should have done.
2. **Classify:** guide gap or sensor gap?
3. **Layer:** pick the lightest layer that fixes it.
4. **Check:** add one check that fails on the old run.
5. **Rerun:** run every check, on every version.
6. **Prune:** run the ablation, and remove what does nothing.

### Steps 1 and 2: Log It and Classify It

An **incident** = any run where the agent did the wrong thing.

Write it down, short and exact. A shared file such as `docs/agent-incidents.md` is enough.

Then ask two questions:

- **Did the agent know the rule?** If not, it is a **guide gap**. Fix it with a rules line or a skill.
- **Did any check catch the mistake?** If not, it is a **sensor gap**. Fix it with a test, a lint rule or a hook.

One incident can be both.

Step through all five frames.

- **Frame 1:** the real transcript. Two steps are marked red.
- **Frame 2:** the rules file already says "never". So it is not a guide gap.
- **Frame 3:** the new check: one hook and two deny lines.
- **Frame 5:** the grader fails on the old harness and passes on the new one.

### Step 3: Pick the Layer

Pick the lightest layer that fixes the gap:

| The gap | The layer |
|---|---|
| A fact that everyone needs, every time | a rules line |
| Steps for one kind of job | a skill |
| A rule that a machine can check | a test or a lint rule, with a message that names the fix |
| A "never" that must not happen even once | a hook or a permission deny |

### Steps 4 and 5: Add a Check, Then Rerun Everything

The new check must fail on the old run.

If it passes on the run that went wrong, it catches nothing.

Then rerun every check on every version. A new rule can break an old job.

Keep the run that went wrong. It becomes a test for your harness.

Every lesson in this course started this way: a failing run, then a fix that makes it pass.

**Simple lesson:** a check that never failed has not proved anything.

## Ablation: Which Parts Carry Weight?

### What Ablation Means

Ablation is a word from research.

You take one part out, run the same tests, and see what changes.

**Load-bearing** = a check fails without this part. It is like a wall that holds up the roof.

If nothing changes, the part can go.

Step through all five frames.

- **Red stages:** a check broke without that part. It stays.
- **Green stages:** nothing broke. It can go.
- **Frame 5:** the .NET project gives a different answer for one part.

### Why These Three Can Go

The React ablation found 3 parts that change nothing:

- **"Read the failing test and the file you plan to change before you edit anything."** In this lab, removing it changed nothing, because the lab agent reads first anyway. A real agent may not. Run a few real tasks without the line before you drop it.
- **"Write clean code. Be careful."** The agent cannot act on it. No check can see it. It is a wish, not a rule.
- **`docs/agent-rules-copy.md`** is a copy of the rules. No rule points to it, so the agent never reads it. Copies also drift apart over time.

Remove one part at a time. Rerun every check after each removal.

Two parts can cover the same job.

Removing one may be fine. Removing both breaks the job.

### Lab Exercise: Make the Harness Lean

You ran the ablation at the top of this lesson. Now act on it:

1. In the rules file, delete the "Read the failing test..." line.
2. Delete the "Write clean code. Be careful." line.
3. Delete `docs/agent-rules-copy.md`.

Click the button, make the changes, then press **Check my harness**.

- **Pass:** every part left is load-bearing, and every check still passes.
- **Fail:** the grader names the check that broke, or the dead part still there.

**Simple lesson:** cut only what the ablation marked "can go".

## Keeping the Harness Small

A small harness is easier for the agent to read. It is also easier for you to trust.

Step through all four frames.

- **Frame 1:** the rules file. Green lines carry weight. Red lines changed nothing.
- **Frame 2:** a copy of the rules that nobody loads.
- **Frames 3 and 4:** the trimmed file, and the same checks passing.

**Lean harness** = a harness where every part is load-bearing.

### The Loop Across the Team

Treat harness fixes like any other code change.

- Give each incident an owner.
- Rerun the ablation after a model upgrade. A new model may not need some lines.
- Run your harness tests in CI (see **Tests and Linters**) when someone edits a rule, a skill or a hook.

**Simple lesson:** a change to the harness is a code change, so test it.

## When Not to Add a Part

Every part of a harness costs something.

| Part | What it costs | Add it when |
|---|---|---|
| A rules line | Reading room for the agent, every run | The agent got a fact wrong |
| A skill | Upkeep when the job changes | The same job keeps coming out different |
| A hook or deny rule | Can block good work if it is too broad | Something must never happen |
| An end-to-end check | Minutes on every run | Green tests hid a broken app |
| An evaluator | A second agent run: time and money | Long or risky work, judged by the builder |
| A subagent or MCP server | More setup, more access to manage | Big output or a system the agent cannot see |

A small or solo repo usually needs only a little:

1. A short rules file with the exact test command.
2. One "never edit" line for the file that matters most.
3. One or two guards, such as no deletes and no switched-off tests.

Add anything more only after a real failure shows you need it. That is the improvement loop.

**Simple lesson:** start small. Let real mistakes tell you what to add.

## TL;DR

1. **Improvement loop** = incident, classify, pick the layer, add a check, rerun, prune.
2. **Guide gap** = the agent did not know the rule. Add a rules line or a skill.
3. **Sensor gap** = nothing caught the mistake. Add a test, a lint rule or a hook.
4. **A good check** = one that fails on the old run.
5. **Ablation** = remove one part at a time and rerun every check.
6. **Load-bearing** = a check fails without it.
7. **Lean harness** = only load-bearing parts. Less context, same checks.

## Common mistakes

| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| No incident log | "We will remember." | Write each incident down. | The same mistake comes back. |
| Every fix is a rules line | "Tell it, and it will obey." | A sensor gap needs a check. | The rule is ignored, and nothing catches it. |
| A check that never failed | "I added a test, so it is fixed." | The new check must fail on the old run. | The check catches nothing. |
| Keep every line | "More rules mean more safety." | Keep only load-bearing parts. | Context fills up and real rules get lost. |
| Copies of the rules | "A backup in docs is safe." | Keep one source of truth. | The copies drift apart and disagree. |
| Remove by feel | "This line looks useless." | Run the ablation. | A load-bearing part is gone, and a check breaks. |

## Check Your Understanding

## Exercises

These exercises use the real ablation from this lesson.

Press **Check** when you are ready. Use **Show Hint** if you get stuck.

## What to Remember

- The loop: incident, classify, pick the layer, add a check, rerun, prune.
- A guide gap needs a guide. A sensor gap needs a sensor. Some incidents need both.
- A new check must fail on the run that went wrong.
- Ablation takes out one part at a time and reruns every check. Load-bearing parts stay.
- A lean harness uses less context, and every line in it has a reason.

## Further Reading

- [Harness engineering for coding agent users, on martinfowler.com](https://martinfowler.com/articles/harness-engineering.html): guides, sensors, and how to improve the harness over time.
- [Skill issue: harness engineering for coding agents, by HumanLayer](https://www.humanlayer.dev/blog/skill-issue-harness-engineering-for-coding-agents): keep instructions short and remove what does not help.
- [Harness engineering: leveraging Codex in an agent-first world, by OpenAI](https://openai.com/index/harness-engineering/): turning each agent mistake into a check.

Next: **Capstone Project**. You build one harness that passes every job in the course, and then you export it.