# Observability and Replay

Course: [Graph Engineering](https://www.devtutorials.ai/learn/graph-engineering) · Section: Graphs in Production
Source: https://www.devtutorials.ai/learn/graph-engineering/graphs-in-production/observability-and-replay

> When a graph run goes wrong, compare it with a run that worked. Find the first node that differs, read per-node tokens and verdicts, and see where the money goes.

## Watch a Run Fail With No Clue Why

Last night's review run went wrong three times:

- a payment ticket was merged without anyone approving it;
- a GPL licence problem was missed;
- one run cost four times as much as usual and fixed nothing.

The final result of each run tells you **that** something broke, not **where**. In a graph, the cause is often several nodes before the place where you notice it.

Open the Inspector and look at the first failed run next to a run that worked on the same kind of ticket.

The merge at the end looks like the problem. It is not. Read the two logs from the top: the audits match, the verifier matches, and then the **router** gave a different answer. Everything after that is an effect.

## TL;DR

- **Log every node**: what it read, what it wrote, its tokens, its time and its verdict.
- To debug, **diff** a failed run against a run that worked. The **first** node that differs is the cause. Later differences are effects.
- Keep run logs as files you can **replay**: step through a past run without paying for it again.
- Read **cost per node**. Cut cost where the tokens actually go.
- Set a **spend cap per node** for the expensive ones, so one node cannot eat the whole budget.

## A Real-World Example

Dev's team has a five-node graph that answers support questions. One week, answers get worse. The team first rewrites the final "answer" prompt. No change. Then they rewrite it again.

Finally someone compares the logs of a bad answer and a good one, node by node. The first difference is in the **search** node: an index update had stopped returning the help-centre pages. The answer node was doing its best with the wrong input. Two days of prompt changes had been fixing the wrong node.

Now every run writes a small log line per node, and the first step of every bug is: diff it against a good run.

## What You Will Learn

- What to log for every node
- How to diff a failed run against a working one
- Why the first difference is the cause, and the rest are effects
- How to read cost per node and cut it where it matters
- How replay saves you from paying for the same run twice

## Cost and Finding the Node That Broke

A graph run is a sequence of node steps. If each step leaves a short log line, you can read the run like a story.

```js runnable title="A per-node log"
const log = [
  { node: "security",    tokens: 3100, seconds: 42,  verdict: "0 issues" },
  { node: "licence",     tokens: 2600, seconds: 35,  verdict: "1 GPL package flagged" },
  { node: "verify",      tokens: 900,  seconds: 4,   verdict: "2 of 2 audits" },
  { node: "route",       tokens: 600,  seconds: 2,   verdict: "high (0.91) -> careful_fix" },
  { node: "careful_fix", tokens: 7400, seconds: 180, verdict: "pass after 3 tries" },
];

for (const l of log) console.log(l.node.padEnd(12) + String(l.tokens).padStart(6) + " tok " + String(l.seconds).padStart(4) + " s  " + l.verdict);
const total = log.reduce((s, l) => s + l.tokens, 0);
console.log("total: " + total + " tokens");
```

Expected output:

```text
security      3100 tok   42 s  0 issues
licence       2600 tok   35 s  1 GPL package flagged
verify         900 tok    4 s  2 of 2 audits
route          600 tok    2 s  high (0.91) -> careful_fix
careful_fix   7400 tok  180 s  pass after 3 tries
total: 14600 tokens
```

Five things per node are enough to start: the **node**, its **inputs** (which fields it read), its **tokens**, its **time** and its **verdict**. In a graph pack from this course, the supervisor appends one such line per node to `graph-state.json`.

### Diff Two Runs

```js runnable title="Find the first node that differs"
const working = [
  ["security", "0 issues"], ["licence", "1 GPL package"], ["verify", "2 of 2"],
  ["route", "high (0.91)"], ["careful_fix", "pass"], ["approve", "waiting"],
];
const failed = [
  ["security", "0 issues"], ["licence", "1 GPL package"], ["verify", "2 of 2"],
  ["route", "low (0.62)"], ["quick_fix", "pass"], ["merge", "merged"],
];

const first = failed.findIndex((row, i) => row[0] !== working[i][0] || row[1] !== working[i][1]);
console.log("first difference: " + failed[first][0] + " (" + working[first][1] + " vs " + failed[first][1] + ")");
console.log("effects after it: " + failed.slice(first + 1).map((r) => r[0]).join(", "));
```

Expected output:

```text
first difference: route (high (0.91) vs low (0.62))
effects after it: quick_fix, merge
```

The router said "low" with a confidence of 0.62. That is the cause. Fixing the merge step, or the quick fixer, would change nothing.

The node where you **notice** a problem is usually not the node that **caused** it. A bad merge, a wrong answer or a high bill is where the trouble shows up. Always walk the log from the top.

### Replay

A **replay** steps through a past run's log without running any node again. You can scrub back and forth, read what each node saw, and compare it with another run. It costs nothing, because it only reads files.

That is what the Run tab in this lab does: every scenario's events are a log you can step through and scrub. Each node shows a live badge with its tokens, steps and tries.

Your own runs are logged too. Run a graph twice, for example before and after a fix. Open the **Inspector**. Under **Your runs**, pick run A and run B. The lab shows the first node where they differ.

### Replay a Log Step by Step

A replay is just a loop over the saved log, with a position you can move:

```js runnable title="Replay to a position"
const log = ["security: 0 issues", "licence: timed out", "verify: 1 of 2 (partial)", "route: high", "careful_fix: pass"];

function replay(log, upTo) {
  return log.slice(0, upTo).map((line, i) => (i + 1) + ". " + line).join("\n");
}

console.log(replay(log, 3));
console.log("(no node ran again: 0 tokens)");
```

Expected output:

```text
1. security: 0 issues
2. licence: timed out
3. verify: 1 of 2 (partial)
(no node ran again: 0 tokens)
```

## Where the Tokens Go

The same log answers a second question: where does the money go?

```js runnable title="Cost by node"
const spend = { careful_fix: 46800, quick_fix: 37200, security: 28000, licence: 28000, route: 12000, verify: 6000 };
const total = Object.values(spend).reduce((a, b) => a + b, 0);

const rows = Object.entries(spend).sort((a, b) => b[1] - a[1]);
for (const [node, t] of rows) console.log(node.padEnd(12) + String(Math.round((t / total) * 100)).padStart(3) + "%");
```

Expected output:

```text
careful_fix  30%
quick_fix    24%
security     18%
licence      18%
route         8%
verify        4%
```

Two lessons from this table:

- **Cut where the tokens are.** Shortening the router's prompt saves almost nothing. Sending one fewer ticket to the careful fixer saves a lot. A better router is often the cheapest way to cut cost.
- **Parallel branches multiply shared context.** Every audit reads the ticket. Ten audits read it ten times. Give each branch only the fields it needs.

### A Spend Cap per Node

A single stuck node can eat the whole run's budget, as Lesson 7 showed. Give the expensive nodes their own `budget:` in `exit:`, and the run a total limit.

```js runnable title="Which node would break a per-node cap?"
const caps = { careful_fix: 12000, quick_fix: 3000 };
const runs = [
  { node: "careful_fix", ticket: 25, tokens: 7400 },
  { node: "careful_fix", ticket: 13, tokens: 30100 },
  { node: "quick_fix",   ticket: 21, tokens: 2100 },
];

for (const r of runs) {
  const over = r.tokens > caps[r.node];
  console.log("#" + r.ticket + " " + r.node + ": " + r.tokens + (over ? "  OVER CAP -> stopped on bound: budget" : "  ok"));
}
```

Expected output:

```text
#25 careful_fix: 7400  ok
#13 careful_fix: 30100  OVER CAP -> stopped on bound: budget
#21 quick_fix: 2100  ok
```

All token numbers in this lab are approximate. They are there to show where cost goes and how it changes, not to predict your bill. Your tool's own usage report gives the real numbers for a real run.

## Lab Exercise: Find the Node That Broke

Open the **Inspector** tab. It shows three failed runs. Each one sits next to a run that worked on the same kind of ticket.

For each failed run:

1. Read both logs from the top.
2. Find the **first** row that differs.
3. Press **This broke** on that row.

You have **2 tries** per run. When all three are done, press **Check my answers**.

The graph pack you export gives your real runs the same log: the supervisor writes one line per node into `graph-state.json`, under `run_log`. Diff two of those logs exactly as you do in this lab.

## Common mistakes

| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| Fixing the last node | "The merge was wrong, so fix the merge." | The first difference is the cause. | You fix an effect; the bug stays. |
| No good run to compare | "I'll read the failed log carefully." | Diff it against a run that worked. | Normal behaviour looks suspicious, and the real cause hides. |
| No per-node log | "The final result is enough." | One line per node. | You cannot tell where it went wrong. |
| Cutting cheap nodes | "Every token counts the same." | Cut where the tokens go. | Effort spent, bill unchanged. |
| Re-running to debug | "Run it again and watch." | Replay the log. | You pay again, and the run may not repeat. |

## Exercises

Each exercise ends with a value on the last line. Press **Check** to compare it with the expected answer.

## What to Remember

- Log node, inputs, tokens, time and verdict for every step.
- Diff a failed run against a good one. The first difference is the cause.
- Replay reads the log, so debugging costs nothing.
- Read cost per node and cut where the tokens go.
- Give expensive nodes their own spend cap.

## Further Reading

- [How to Implement Graph Engineering, by QA Wolf](https://www.qawolf.com/blog/how-to-implement-graph-engineering-5-composable-patterns): tracking token budgets and spotting rising cost with a flat score.
- [Building Effective Agents, by Anthropic](https://www.anthropic.com/engineering/building-effective-agents): keep agent systems simple and transparent, so you can see each step.

Next: **Human Gates**. A payment change is merged with no one looking, and you put a person on the risky path, with the evidence they need.