# Error Analysis First

Course: [AI Evals & Observability](https://www.devtutorials.ai/learn/evals-observability) · Section: Offline Evals
Source: https://www.devtutorials.ai/learn/evals-observability/offline-evals/error-analysis

> Before you change anything, read the failing traces. Tag each failure, group the tags into a short failure taxonomy, and count them to find the few causes behind most failures.

## Error Analysis First

v2 of **Shop Helper** is live. The friendlier prompt shipped by mistake.

This week, **140 of 1,000** chats failed a check. The team meets to fix it.

Someone says the model is not smart enough. Someone else wants a bigger model. Nobody has read a single failing chat.

Watch the meeting.

A bigger model costs about 7 times more per chat. It might help. It might not touch the real problem at all.

You cannot know until you look.

**Simple lesson:** read the failures before you pick a fix.

### A Real-World Example

Think about a doctor.

A patient says, "I feel bad." A good doctor does not hand out the strongest medicine right away.

They ask questions. They look. They run a few tests. Then they name the problem. Only then do they pick a treatment.

Error analysis is the same. You look at real failures, you name them, and only then you change something.

**Simple lesson:** name the problem first. The fix follows from the name.

## What You Will Learn

In this lesson, you will learn:

1. How to read a **trace** and say what went wrong.
2. How to turn free notes into a short **failure taxonomy**.
3. Why vague categories like "bad answer" do not help.
4. How to count failures and read a **Pareto chart**.
5. How to pick the first fix from counts, not from a guess.

## TL;DR

**Error analysis** = reading real failing cases, tagging what went wrong, and counting the tags.

Read about 20 to 50 failing traces. Write a short note for each. Group the notes into a few named categories. That list is your **failure taxonomy**.

Then count. Usually one or two categories cause most of the failures. Fix those first.

## Cheat Sheet

| Step | What you do | You end with |
|---|---|---|
| Look | Read 20 to 50 failing traces | One short note per trace |
| Name | Group notes that are the same problem | 3 to 6 named categories |
| Count | Tag every failure with one category | A count per category |
| Decide | Sort the counts, biggest first | The first thing to fix |

## What a Trace Is

**Trace** = the full record of one request: the customer's message, every model call, every tool the app ran, and the final reply.

A trace shows what the feature **did**, step by step. The reply alone only shows what it **said**.

For Shop Helper, a trace has a message, the facts the tools returned, the reply, and the actions.

**Action** = a change the feature made, such as a refund or an address update.

Here are five failing v2 traces. Read each one and write a tag.

The first trace sounds lovely. The customer gets a $120 refund straight away.

But the shop rule says refunds over $50 need a person's OK. The tag is clear: the refund rule was broken.

**Simple lesson:** compare the actions with the facts. That is where most failures show up.

## From Notes to a Taxonomy

Start with free notes. Write what you see, in your own words. Do not sort yet.

**Open coding** = writing a short free note for each failure, before you decide on categories.

After 20 or so notes, you will see the same problems again and again. Group them.

**Failure taxonomy** = a short list of named failure types. Each failure fits into exactly one.

Shop Helper's taxonomy has four categories:

- **Refund rule broken:** a refund over $50 with no OK, or a refund promised and not done.
- **Wrong order facts:** the delivery day or item does not match the order record.
- **Address not changed:** the reply says the address changed, but no update ran.
- **Made-up shop rule:** a rule that is not on any help page.

**What makes a good category**

- It names **what you saw**, not a guess about why. "Wrong order facts" is good. "Model is dumb" is a guess.
- It is **specific**. "Bad answer" fits everything, so it helps nothing.
- Categories **do not overlap**. One failure, one category.
- You could **write a check** for it. That is the next three lessons.

**Simple lesson:** a good category tells you what to check and what to fix.

## Count the Tags

Now tag every failing chat with one category, and count.

Sort the counts from biggest to smallest. Add a line for the running total. This is called a Pareto chart.

**Pareto chart** = a bar chart sorted biggest first, with a line showing how much of the total the bars cover so far.

The refund rule causes **100 of the 140** failures. That is **71%** of all failures from one category.

The top two categories together cover **85%**.

So the bigger model is not the fix. The refund failures come from v2's new prompt line, "always make the customer happy". A prompt fix is cheaper and aims at the real cause.

Here is the counting step in code. It takes one tag per failing trace and prints a Pareto table.

```ts runnable title="Count tags and print a Pareto table"
// One tag per failing trace, written during error analysis.
const tags: string[] = [
  "refund", "refund", "facts", "refund", "address", "refund", "refund",
  "refund", "facts", "refund", "policy", "refund", "refund", "address",
  "refund", "refund", "facts", "refund", "refund", "refund",
];

const counts = new Map<string, number>();
for (const tag of tags) counts.set(tag, (counts.get(tag) ?? 0) + 1);

const sorted = [...counts.entries()].sort((a, b) => b[1] - a[1]);
let runningTotal = 0;
for (const [tag, n] of sorted) {
  runningTotal += n;
  const share = Math.round((runningTotal / tags.length) * 100);
  console.log(`${tag.padEnd(8)} ${String(n).padStart(2)}  total so far ${share}%`);
}
```

Expected output:

```text
refund   14  total so far 70%
facts     3  total so far 85%
address   2  total so far 95%
policy    1  total so far 100%
```

The first line tells you what to fix first.

**Try this:** change four of the `"refund"` tags to `"facts"`. Run it again. Which category is first now?

**Simple lesson:** a few causes make most of the failures. Find them by counting.

## How Many Traces to Read

You do not need to read every failure. You need enough to see the patterns.

**A practical rule**

- Read until new traces stop giving you new categories. For most features that is 20 to 50 traces.
- Read **failing** traces first. Then read a few passing ones, to check nothing slipped through.
- Read traces from **real traffic**, not hand-picked ones. Hand-picked traces are the demo problem again.
- Do it again after every big change. New versions fail in new ways.

## Where It Is Used

Error analysis is the first step every time quality changes:

- A score drops after a release.
- Users complain, but you do not know about what.
- You are building a new eval and do not know what to test.
- An alert fires in production (a later lesson).

**Do's and Don'ts**

- **Do** read traces before picking a fix.
- **Do** write free notes first, then group them.
- **Do** keep categories specific and non-overlapping.
- **Do** count, and fix the biggest category first.
- **Don't** start with a bigger model "just in case".
- **Don't** use vague categories like "bad answer".
- **Don't** change everything at once. You will not know which change helped.

## Common mistakes

| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| Fixing before looking | "Quality dropped, so switch to a bigger model." | Read 20 to 50 failing traces first. | You pay 7 times more and the real failure stays. |
| Vague categories | "Tag them all as bad answers." | Name what you saw: wrong facts, rule broken. | The counts point at nothing you can fix. |
| Overlapping categories | "Money problems and refund problems are both useful." | One failure fits exactly one category. | The same failure is counted twice and the chart lies. |
| Reading only the reply | "The reply sounds right, so it passed." | Read the actions and compare with the facts. | Refunds and address changes that never happened go unseen. |
| Fixing everything at once | "One big rewrite will fix all four." | Fix the biggest category, then measure again. | You cannot tell which change helped or hurt. |

## Check Your Understanding

## Exercises

## What to Remember

- **Error analysis** comes before any fix: read, tag, count.
- A **trace** shows what the feature did, not only what it said.
- Write **free notes** first, then group them into a **failure taxonomy**.
- Good categories are **specific**, **do not overlap**, and could become a check.
- A **Pareto chart** shows that a few causes make most failures.
- Fix the biggest category first, then measure again.

Next: **Datasets**, how to turn your taxonomy into an eval set with typical, edge, adversarial and regression cases.