# Debugging from Traces

Course: [AI Evals & Observability](https://www.devtutorials.ai/learn/evals-observability) · Section: Observability and Production
Source: https://www.devtutorials.ai/learn/evals-observability/production/debugging-from-traces

> The quality alert fires on day 12. Follow it to the failing traces, compare a good trace with a bad one, find the stale search index, fix it, and add a regression case so it never slips through again.

## Debugging from Traces

It is **day 12**. The quality alert for Help Centre Q&A fires at 08:00.

The on-call developer has a guess straight away. "The provider must have changed the model again."

So they spend the morning rewriting the prompt. They add "Always quote the latest prices." The score does not move.

The model was never the problem. Watch what the traces showed all along.

The answer was sitting in one span. The page copy in the search index was older than the page on the site.

**Simple lesson:** do not guess the cause. Read the traces.

### A Real-World Example

Think about a car with a warning light.

A good mechanic does not start by changing the engine oil because it fixed the last car. They plug in a reader and look at the error codes.

Then they compare with a car that works. Whatever is different is the first suspect.

Traces are your error codes. A passing trace is your working car.

**Simple lesson:** compare a broken run with a working one, and look at what is different.

## What You Will Learn

In this lesson, you will learn:

1. The debugging loop: alert, traces, root cause, regression case, fix, re-run.
2. How to group failing traces to find a pattern.
3. How to compare a good trace with a bad one.
4. How to name a **root cause** from span attributes.
5. How to write a **regression case** that closes the loop.

## TL;DR

**Root cause** = the first thing that went wrong, which caused everything after it.

**Regression case** = a saved test case that reproduces a bug, kept in the eval set so the bug cannot come back unnoticed.

Start from the alert. Filter to failing traces. Group them. Compare one with a passing trace. The attribute that differs is your first suspect.

Add a regression case that fails now. Fix the cause. Run the evals again. The case should pass.

## Cheat Sheet

| Step | Question | Where to look |
|---|---|---|
| Alert | What crossed the line, and when? | The alert and its runbook |
| Filter | Which chats failed in that window? | Traces with a failing eval score |
| Group | What do the failures share? | Counts by page, model, version, tool |
| Compare | What differs from a passing chat? | Span attributes side by side |
| Fix | What change removes the cause? | Data, code, prompt or config |
| Regression case | Will we notice if it comes back? | The eval set |

## The Debugging Loop

An alert tells you **that** something is wrong. It does not tell you **why**.

The debugging loop gets you from the alert to the cause, step by step:

1. **Read the alert** and open its runbook.
2. **Filter** to failing traces from the alert window.
3. **Group** the failures by page, model, version and tool.
4. **Compare** one failing trace with one passing trace.
5. **Name the root cause.**
6. **Add a regression case** that fails now.
7. **Fix** the cause.
8. **Re-run** the evals and watch the alert close.

Steps 6 and 7 can feel backwards. Add the case first. If you fix first, you cannot prove the case really tests the bug.

**Simple lesson:** alert, traces, cause, case, fix, re-run.

## Group Before You Read

There may be hundreds of failing chats. Reading them one by one takes all day.

Group them first. In this incident, **63 of the 200** eval questions are about the two pages the shop changed. While the index is stale, **57 of them** fail. Everything else looks normal.

That pattern points straight at those two pages.

Grouping is easy to write in code. This cell counts six failing traces by each attribute.

```ts runnable title="Group failing traces by attribute"
// Six failing traces from the alert window, reduced to the attributes
// that could explain them.
type Failing = { model: string; version: string; page: string; indexedOn: number; updatedOn: number };

const failing: Failing[] = [
  { model: "model-s", version: "v1", page: "express", indexedOn: 1, updatedOn: 12 },
  { model: "model-s", version: "v1", page: "refunds", indexedOn: 1, updatedOn: 12 },
  { model: "model-s", version: "v1", page: "express", indexedOn: 1, updatedOn: 12 },
  { model: "model-s", version: "v1", page: "express", indexedOn: 1, updatedOn: 12 },
  { model: "model-s", version: "v1", page: "refunds", indexedOn: 1, updatedOn: 12 },
  { model: "model-s", version: "v1", page: "gift-cards", indexedOn: 1, updatedOn: 1 },
];

function countBy(label: string, pick: (t: Failing) => string): void {
  const counts = new Map<string, number>();
  for (const t of failing) counts.set(pick(t), (counts.get(pick(t)) ?? 0) + 1);
  const parts = [...counts].map(([value, n]) => `${value} ${n}`);
  console.log(`${label.padEnd(10)} ${parts.join(", ")}`);
}

countBy("model", (t) => t.model);
countBy("version", (t) => t.version);
countBy("page", (t) => t.page);
countBy("stale copy", (t) => (t.indexedOn < t.updatedOn ? "yes" : "no"));
```

Expected output:

```text
model      model-s 6
version    v1 6
page       express 3, refunds 2, gift-cards 1
stale copy yes 5, no 1
```

Every failure has the same model and version. So the model and the prompt cannot explain why only some chats fail.

Five of six have a stale page copy. The one gift-card failure is ordinary noise: v1 sometimes fails anyway.

**Try this:** change the gift-card trace to `updatedOn: 12`. Run it again. Now every failure has a stale copy.

## Compare a Good Trace With a Bad One

A trace from a passing chat in the same hour is your control. Anything the two traces share is not the cause.

The model is the same. The prompt version is the same. Only the page dates differ.

This works because the spans carry useful attributes: the model, the app version, and which page was retrieved and when. Lesson 8 showed how to add them.

**Simple lesson:** a trace is only as useful as the attributes you put on it.

**About this incident**

- On day 12 the shop changed two help pages. Express delivery now costs $12, and refunds now take 3 working days.
- The pages were edited, but the search index still held copies from day 1. That is a **stale index**.
- In this lesson the stale traces are made from the output bank. The lab's Traces tab shows the normal traces, so you can see where the page attributes live.

## Fix It and Close the Loop

The root cause is a stale search index. Rewriting the prompt could never fix it.

The fix has two parts. Re-index the pages now. Then make sure it cannot happen again: re-index any page within minutes of an edit, and alert if a page is newer than its indexed copy.

Before fixing, save a failing chat as a regression case.

A good regression case has two properties:

- It **fails before** the fix. Otherwise it does not test this bug.
- It **passes after** the fix. Otherwise it tests something else.

Add one for each thing that broke. Here, one express question and one refund question.

**A regression case is a promise**

Once it is in the eval set, the CI gate from lesson 7 runs it on every change.

If someone breaks re-indexing next month, the pull request fails before it ships. Nobody has to remember this incident.

## Where It Is Used

- **After every alert** that turns out to be real.
- **After a customer complaint**, using the complained-about chat.
- **After a thumbs-down chat** that shows a new kind of failure.
- **In a post-incident review**, to show what changed and what now guards it.

**Do's and Don'ts**

- **Do** start from the traces, not from a guess.
- **Do** group failures before reading them one by one.
- **Do** compare with a passing trace from the same window.
- **Do** add the regression case before the fix.
- **Don't** change the prompt until you know the prompt is the cause.
- **Don't** close the alert until the evals pass again.

## Common mistakes

| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| Guessing the cause | "It is always the model." | Read and group the failing traces first. | Hours are spent fixing the wrong thing. |
| Reading one trace at a time | "I will read them all." | Group failures by page, model and version. | You see details and miss the pattern. |
| No control trace | "This trace looks wrong, so this is the cause." | Compare with a passing trace from the same hour. | You blame things the good chats share too. |
| Fixing before adding a case | "I will add a test later." | Add a regression case that fails now. | Nobody can prove the case tests the bug. |
| Fixing the symptom | "Add 'quote the latest price' to the prompt." | Fix the stale index and the process behind it. | The bug returns with the next page edit. |

## Check Your Understanding

## Exercises

## What to Remember

- An alert says **that** something is wrong. Traces say **why**.
- **Group** the failures, then **compare** a failing trace with a passing one.
- The attribute that differs is your first suspect. That is the **root cause**.
- Add a **regression case** that fails now, then fix, then re-run.
- Fix the cause and the process behind it, not the symptom.

Next: **Experiments for Prompts and Models**, how to test a new prompt or model on live traffic before everyone gets it.