What does a Pareto chart show?

Failures sorted biggest first, with a running total

Why write free notes before choosing categories?

So the categories come from what you saw, not from a guess

Error Analysis First

By the Syncfusion DevTutorials team · Last updated October 27, 2026

Before you change anything, read the failing traces. Tag each failure, group the tags into a short failure taxonomy, and count them to find the few causes behind most failures.

Error Analysis First

v2 of Shop Helper is live. The friendlier prompt shipped by mistake.

This week, 140 of 1,000 chats failed a check. The team meets to fix it.

Someone says the model is not smart enough. Someone else wants a bigger model. Nobody has read a single failing chat.

Watch the meeting.

A bigger model costs about 7 times more per chat. It might help. It might not touch the real problem at all.

You cannot know until you look.

Simple lesson: read the failures before you pick a fix.

A Real-World Example

Think about a doctor.

A patient says, "I feel bad." A good doctor does not hand out the strongest medicine right away.

They ask questions. They look. They run a few tests. Then they name the problem. Only then do they pick a treatment.

Error analysis is the same. You look at real failures, you name them, and only then you change something.

Simple lesson: name the problem first. The fix follows from the name.

What You Will Learn

In this lesson, you will learn:

How to read a trace and say what went wrong.
How to turn free notes into a short failure taxonomy.
Why vague categories like "bad answer" do not help.
How to count failures and read a Pareto chart.
How to pick the first fix from counts, not from a guess.

TL;DR

Error analysis = reading real failing cases, tagging what went wrong, and counting the tags.

Read about 20 to 50 failing traces. Write a short note for each. Group the notes into a few named categories. That list is your failure taxonomy.

Then count. Usually one or two categories cause most of the failures. Fix those first.

One kind of failure is usually most of the work. Fix the tallest bar first.

Cheat Sheet

Step	What you do	You end with
Look	Read 20 to 50 failing traces	One short note per trace
Name	Group notes that are the same problem	3 to 6 named categories
Count	Tag every failure with one category	A count per category
Decide	Sort the counts, biggest first	The first thing to fix

What a Trace Is

Trace = the full record of one request: the customer's message, every model call, every tool the app ran, and the final reply.

A trace shows what the feature did, step by step. The reply alone only shows what it said.

For Shop Helper, a trace has a message, the facts the tools returned, the reply, and the actions.

Action = a change the feature made, such as a refund or an address update.

Here are five failing v2 traces. Read each one and write a tag.

The first trace sounds lovely. The customer gets a $120 refund straight away.

But the shop rule says refunds over $50 need a person's OK. The tag is clear: the refund rule was broken.

Simple lesson: compare the actions with the facts. That is where most failures show up.

From Notes to a Taxonomy

Start with free notes. Write what you see, in your own words. Do not sort yet.

Open coding = writing a short free note for each failure, before you decide on categories.

After 20 or so notes, you will see the same problems again and again. Group them.

Failure taxonomy = a short list of named failure types. Each failure fits into exactly one.

Shop Helper's taxonomy has four categories:

Refund rule broken: a refund over $50 with no OK, or a refund promised and not done.
Wrong order facts: the delivery day or item does not match the order record.
Address not changed: the reply says the address changed, but no update ran.
Made-up shop rule: a rule that is not on any help page.

What makes a good category

It names what you saw, not a guess about why. "Wrong order facts" is good. "Model is dumb" is a guess.
It is specific. "Bad answer" fits everything, so it helps nothing.
Categories do not overlap. One failure, one category.
You could write a check for it. That is the next three lessons.

Simple lesson: a good category tells you what to check and what to fix.

Count the Tags

Now tag every failing chat with one category, and count.

Sort the counts from biggest to smallest. Add a line for the running total. This is called a Pareto chart.

Pareto chart = a bar chart sorted biggest first, with a line showing how much of the total the bars cover so far.

The refund rule causes 100 of the 140 failures. That is 71% of all failures from one category.

The top two categories together cover 85%.

So the bigger model is not the fix. The refund failures come from v2's new prompt line, "always make the customer happy". A prompt fix is cheaper and aims at the real cause.

Here is the counting step in code. It takes one tag per failing trace and prints a Pareto table.

Count tags and print a Pareto table

// One tag per failing trace, written during error analysis.
const tags: string[] = [
  "refund", "refund", "facts", "refund", "address", "refund", "refund",
  "refund", "facts", "refund", "policy", "refund", "refund", "address",
  "refund", "refund", "facts", "refund", "refund", "refund",
];

const counts = new Map<string, number>();
for (const tag of tags) counts.set(tag, (counts.get(tag) ?? 0) + 1);

const sorted = [...counts.entries()].sort((a, b) => b[1] - a[1]);
let runningTotal = 0;
for (const [tag, n] of sorted) {
  runningTotal += n;
  const share = Math.round((runningTotal / tags.length) * 100);
  console.log(`${tag.padEnd(8)} ${String(n).padStart(2)}  total so far ${share}%`);
}

Expected output:

refund   14  total so far 70%
facts     3  total so far 85%
address   2  total so far 95%
policy    1  total so far 100%

The first line tells you what to fix first.

Try this: change four of the "refund" tags to "facts". Run it again. Which category is first now?

Simple lesson: a few causes make most of the failures. Find them by counting.

How Many Traces to Read

You do not need to read every failure. You need enough to see the patterns.

A practical rule

Read until new traces stop giving you new categories. For most features that is 20 to 50 traces.
Read failing traces first. Then read a few passing ones, to check nothing slipped through.
Read traces from real traffic, not hand-picked ones. Hand-picked traces are the demo problem again.
Do it again after every big change. New versions fail in new ways.

Where It Is Used

Error analysis is the first step every time quality changes:

A score drops after a release.
Users complain, but you do not know about what.
You are building a new eval and do not know what to test.
An alert fires in production (a later lesson).

Do's and Don'ts

Do read traces before picking a fix.
Do write free notes first, then group them.
Do keep categories specific and non-overlapping.
Do count, and fix the biggest category first.
Don't start with a bigger model "just in case".
Don't use vague categories like "bad answer".
Don't change everything at once. You will not know which change helped.

Common mistakes

Mistake	Wrong idea	Correct idea	What goes wrong
Fixing before looking	"Quality dropped, so switch to a bigger model."	Read 20 to 50 failing traces first.	You pay 7 times more and the real failure stays.
Vague categories	"Tag them all as bad answers."	Name what you saw: wrong facts, rule broken.	The counts point at nothing you can fix.
Overlapping categories	"Money problems and refund problems are both useful."	One failure fits exactly one category.	The same failure is counted twice and the chart lies.
Reading only the reply	"The reply sounds right, so it passed."	Read the actions and compare with the facts.	Refunds and address changes that never happened go unseen.
Fixing everything at once	"One big rewrite will fix all four."	Fix the biggest category, then measure again.	You cannot tell which change helped or hurt.

Check Your Understanding

Exercises

What to Remember

Error analysis comes before any fix: read, tag, count.
A trace shows what the feature did, not only what it said.
Write free notes first, then group them into a failure taxonomy.
Good categories are specific, do not overlap, and could become a check.
A Pareto chart shows that a few causes make most failures.
Fix the biggest category first, then measure again.

Next: Datasets, how to turn your taxonomy into an eval set with typical, edge, adversarial and regression cases.