# Evaluator-Optimizer

Course: [Graph Engineering](https://www.devtutorials.ai/learn/graph-engineering) · Section: Graph Foundations
Source: https://www.devtutorials.ai/learn/graph-engineering/graph-foundations/evaluator-optimizer

> A fixer and a checker go round until the work passes. Give the loop real exits for oscillation, no progress, a flat score and a budget, report a stop on a bound clearly, then export your first graph pack.

## Watch a Fixer Go Back and Forth

A failing test checks how prices are rounded. A **fixer** agent changes the code. A **checker** runs the tests. If they fail, the fixer tries again. That is the evaluator-optimizer pattern, and it is the loop you built in the Loop course.

This fixer has one stop rule: 30 tries. Run it and read the tries.

Try 1 rounds half down: one test fails. Try 2 rounds half up: another test fails. Try 3 rounds half down again. It is going back and forth between two broken fixes, and it will do that 30 times. Then the result says "failed", and nobody can tell what happened.

In this lesson you give the loop exits that notice this early, and report it clearly.

## TL;DR

- **Evaluator-optimizer**: a fixer makes a change, a checker scores it, and they go round until a clear pass condition is met.
- It works only when the pass condition is **mechanical**: a test command, a schema, a number.
- It fails when the exits are weak. Four exits catch the common ways it gets stuck:
  - `oscillation`: going back and forth, A → B → A;
  - `no_progress`: the same diff and the same error again;
  - `flat_score`: several tries with no real gain;
  - `budget`: a token limit for this node.
- Keep `max_iterations` as a small backstop, not the main exit.
- A stop on a bound is **not** a failure. Report it as "stopped on bound" and send it to a person.

## A Real-World Example

Ana's team runs a fixer loop on flaky end-to-end tests. Every morning the bill shows one loop that ran 40 times overnight. Each try changed a wait time from 500 ms to 1000 ms and back.

The logs only said `FAILED`. A developer spent an hour each time working out that the code was fine and the loop was stuck.

They added two exits: stop if a try repeats a try from two steps before, and stop if the same error appears twice in a row. They also changed what the loop reports: "stopped on bound: oscillation, here are both diffs". The overnight cost dropped, and the morning message now says exactly where to look.

## What You Will Learn

- When evaluator-optimizer helps, and what it needs
- Four exits: oscillation, no progress, flat score, budget
- Why the turn cap is a backstop, not a plan
- How to report a stop on a bound and escalate it
- How to export your graph as a graph pack for your tool

## The Pattern

The loop has two roles. The **optimizer** (the fixer) changes the work. The **evaluator** (the checker) scores it against a pass condition. In this course, both live in one node: an `agent` node with an edge back to itself and an `exit:` block.

```yaml
nodes:
  - { id: fix, kind: agent, on_bound: escalate, reads: [ticket, last_error], writes: [diff, last_error],
      exit: { checker: pnpm test, max_iterations: 10, oscillation: true, no_progress: 2, flat_score: 3, budget: 20000 } }
edges:
  - { from: fix, to: fix }
```

It is the right pattern when you can state the pass condition before you start, and a program can check it. "All tests pass" works. "Make the code nicer" does not.

## Four Exits

### Oscillation: A → B → A

The fixer swaps between two broken fixes. Each one fixes the other's test and breaks its own.

```js runnable title="Spot oscillation"
const tries = ["round half down", "round half up", "round half down", "round half up"];

function oscillates(list) {
  for (let i = 2; i < list.length; i++) {
    if (list[i] === list[i - 2] && list[i] !== list[i - 1]) return "oscillation at try " + (i + 1);
  }
  return "no oscillation";
}

console.log(oscillates(tries));
```

Expected output:

```text
oscillation at try 3
```

### No Progress: The Same Thing Again

The fixer makes the same change and gets the same error. Trying a third time will not help.

```js runnable title="Spot no progress"
const tries = [
  { diff: "add null check", error: "test_empty_cart" },
  { diff: "add null check", error: "test_empty_cart" },
  { diff: "add null check", error: "test_empty_cart" },
];

let same = 1;
for (let i = 1; i < tries.length; i++) {
  same = tries[i].diff === tries[i - 1].diff && tries[i].error === tries[i - 1].error ? same + 1 : 1;
  if (same >= 2) { console.log("no progress at try " + (i + 1)); break; }
}
```

Expected output:

```text
no progress at try 2
```

### Flat Score, Rising Cost

The sneakiest one. Every try is different, so it looks busy. But the score does not really improve, and each try reads more context and costs more.

```js runnable title="Spot a flat score"
const scores = [61, 62, 61, 62, 61];
const costs = [1500, 1900, 2300, 2700, 3100];
const FLAT = 3;

for (let i = FLAT; i < scores.length; i++) {
  const before = Math.max(...scores.slice(0, i - FLAT + 1));
  const recent = Math.max(...scores.slice(i - FLAT + 1, i + 1));
  if (recent <= before + 1) {
    const spent = costs.slice(0, i + 1).reduce((a, b) => a + b, 0);
    console.log("flat score at try " + (i + 1) + ", spent " + spent + " tokens");
    break;
  }
}
```

Expected output:

```text
flat score at try 4, spent 8400 tokens
```

### Budget

`budget` is a token limit for this node. It always stops a runaway, but often late. Run the same fixer with **only** a budget:

The budget worked, but only after paying for many useless tries. The other three exits stop it much earlier, because they notice **why** it is stuck. Keep the budget as a safety net under them.

`max_iterations: 30` is not a stop rule. It is the point where you have already paid for 30 tries. Keep the cap small (12 or less), and let the real exits do the work.

## Stopped on a Bound Is Not Failed

When an exit fires, the loop did its job: it noticed it was stuck and stopped. That is very different from "the code is broken" or "the tests are wrong".

Set `on_bound: escalate` on the fixer. Then the run is reported as **"stopped on bound: oscillation"** (or the exit that fired). A person gets the evidence: the last diffs, the errors and the score history.

```js runnable title="Report the stop clearly"
function report(stop, evidence) {
  if (stop === "pass") return "fixed";
  if (stop === "error") return "failed: " + evidence;
  return "stopped on bound: " + stop + " -> sent to a person with " + evidence;
}

console.log(report("pass", ""));
console.log(report("oscillation", "both diffs and both errors"));
console.log(report("error", "the test runner crashed"));
```

Expected output:

```text
fixed
stopped on bound: oscillation -> sent to a person with both diffs and both errors
failed: the test runner crashed
```

In Claude Code and Codex, `/goal` runs a loop until a condition is met (in Cursor it is rolling out in the CLI). In Claude Code, `maxTurns` in a subagent's front matter caps its turns. Write the other exits in plain words in the fixer's instructions: "If your change is the same as two tries ago, stop and report oscillation."

### All Exits Together

Here are the exits working together on one stuck run. The first one that fires stops the loop:

```js runnable title="The first exit that fires"
const tries = [
  { diff: "A", error: "e1", score: 60, tokens: 2500 },
  { diff: "B", error: "e2", score: 58, tokens: 2500 },
  { diff: "A", error: "e1", score: 60, tokens: 2500 },
];
const BUDGET = 20000, CAP = 10;

let spent = 0, stop = "";
for (let i = 0; i < tries.length && !stop; i++) {
  spent += tries[i].tokens;
  if (i >= 2 && tries[i].diff === tries[i - 2].diff && tries[i].diff !== tries[i - 1].diff) stop = "oscillation";
  else if (i > 0 && tries[i].diff === tries[i - 1].diff && tries[i].error === tries[i - 1].error) stop = "no_progress";
  else if (spent >= BUDGET) stop = "budget";
  else if (i + 1 >= CAP) stop = "max_iterations";
}
console.log("stopped on bound: " + stop + " after " + spent + " tokens");
```

Expected output:

```text
stopped on bound: oscillation after 7500 tokens
```

## Lab Exercise: Write the Exits

Open the graph file. Change the `fix` node so that:

1. Its `exit:` has `oscillation: true`, `no_progress: 2`, `flat_score: 3` and a `budget` (for example `20000`).
2. `max_iterations` is 12 or less.
3. It has `on_bound: escalate`.

Then press **Check my graph**. In each of the **3 versions** the fixer gets stuck in a different way: oscillation, no progress, or a flat score with rising cost. Each run must stop under 10,000 tokens and be reported as "stopped on bound".

## Your First Graph Pack

You now have every part of a working graph: nodes, a state contract, a handoff, a router, joins, caps and exits. Time to take one home.

Open the **Export** tab. It turns the graph file you are looking at into files your tool can follow:

- `graph.md`: the topology, written as rules the main agent follows;
- one **subagent file per node**, in your tool's format (`.claude/agents/`, `.codex/agents/` or `.cursor/agents/`);
- `state-contract.yaml`: every field, its type and its one writer;
- a **supervisor** subagent that checks the state file between nodes and stops the graph if something drifts;
- `GRAPH-README.md`: an optional checklist for a first real run.

The smallest useful pack is two nodes: an **executor** that does the work round by round, and a **supervisor** that checks it from outside. The open-source graphkit skill uses the same shape. Each round, the executor writes what it did to a log file. A supervisor with a clean context reads that file, spots when the work goes off track, and writes fixes.

None of the three tools has a "graph" engine. The exported graph works because the main agent follows `graph.md` step by step and calls one subagent per node, with the state in a file. That is why the course teaches graphs as plain files.

## Common mistakes

| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| Turn cap only | "30 tries is a stop rule." | Exits that notice why it is stuck. | You pay for 30 useless tries. |
| Budget only | "The budget will catch it." | Budget is the net under the real exits. | It stops, but late and expensive. |
| No oscillation exit | "Each try is different." | A → B → A is a loop inside the loop. | Two broken fixes swap forever. |
| "Failed" for everything | "It didn't pass, so it failed." | "Stopped on bound: …" plus evidence. | Someone starts the investigation from zero. |
| Soft pass condition | "Make it better." | A test, a schema or a number. | The loop never knows it has finished. |

## Exercises

Each exercise ends with a value on the last line. Press **Check** to compare it with the expected answer.

## What to Remember

- Evaluator-optimizer: a fixer and a checker go round against a mechanical pass condition.
- Four exits catch the common stuck states: oscillation, no progress, flat score, budget.
- The turn cap is a small backstop, not the plan.
- A stop on a bound is not a failure. Report it with evidence and escalate it.
- The Export tab turns any graph file into a graph pack for your tool.

## Further Reading

- [How to Implement Graph Engineering, by QA Wolf](https://www.qawolf.com/blog/how-to-implement-graph-engineering-5-composable-patterns): evaluator-optimizer and the four missing exit conditions.
- [Building Effective Agents, by Anthropic](https://www.anthropic.com/engineering/building-effective-agents): the evaluator-optimizer workflow and when it fits.
- [graphkit, by levi-qiao](https://github.com/levi-qiao/graphkit): a framework-free executor and supervisor graph, kept in Markdown files.
- The **Stop Rules** lesson in the Loop Engineering course: the stop rules this lesson builds on.

Next: **Composing Patterns**. Two audits feed a fixer with no verifier, router or gate, and you compose a review graph that meets a scorecard.