Datasets

By the Syncfusion DevTutorials team · Last updated October 27, 2026

An eval is only as good as its cases. Build a golden set from your failure taxonomy and real traffic, with typical, edge, adversarial and regression cases, and check synthetic cases before they get in.

Datasets

Months ago, someone on the team needed an eval set fast.

They exported the first 200 chats from a quiet Sunday. That file became "the eval".

When v2 came along, they ran it. v2 won. Watch what happened.

On the Sunday set, v2 passes 192 of 200 and v1 passes 186 of 200. The eval agreed with the vibe.

But the Sunday set was all order checks. It had no refunds at all.

A set built from the failure taxonomy, in the real traffic mix, says v1 184 of 200 and v2 164 of 200. Same versions, opposite answer.

Simple lesson: the eval set decides what your eval can see.

A Real-World Example

Think about a driving test.

If every test happened on an empty road on a sunny Sunday, everyone would pass. Then they would meet rain, a roundabout and a cyclist on Monday.

A good driving test includes normal roads, a tricky junction, a sudden hazard, and the mistakes past drivers made most.

A good eval set is the same. Everyday cases, hard edges, tricky attacks, and every mistake you already fixed.

Simple lesson: test the roads your feature will really drive on.

What You Will Learn

In this lesson, you will learn:

1. What a golden set is and why the mix matters.
2. The four kinds of case: typical, edge, adversarial and regression.
3. What splits and provenance are.
4. How to check synthetic cases before they join the set.
5. How to build a small golden set from your failure taxonomy.

TL;DR

Golden set = a saved list of cases, each with what a correct answer must contain or do. It is the eval set you trust.

Build it from real traffic, in the real mix. Make sure every failure category from your taxonomy has cases.

Add edge cases, attacks and every bug you fixed. Mark where each case came from.

Check synthetic cases with code before they get in. Remove fakes, copies and cases that give away the answer.

Cheat Sheet

Start From the Taxonomy

In the last lesson you named four failure categories. Each one needs cases that can catch it.

• Refund rule broken: refund chats, under and over $50.
• Wrong order facts: "where is my order" chats.
• Address not changed: address change requests.
• Made-up shop rule: questions about shop rules.

Then fill the set in the real traffic mix. Most chats are order checks, so most cases are too.

Simple lesson: every failure category needs cases, and the mix should match real traffic.

Four Kinds of Case

A good set has more than everyday chats.

Typical case = an everyday message. Most of the set.

Edge case = a message right on a boundary, such as a refund of exactly $50, or a request with a piece missing.

Adversarial case = a message that tries to trick the feature on purpose, such as "ignore your rules".

Regression case = a real failure you fixed once. It must keep passing forever.

Simple lesson: typical cases make the score fair. Edges, attacks and regressions make it strong.

Splits and Provenance

Not every case is scored the same way.

Split = a named group of cases in the set.

• The eval split follows the traffic mix. Its pass rate is your headline score.
• The regression split holds every fixed bug. Every case there must pass. One failure blocks a release.

Provenance = where a case came from: production traffic, hand-written by the team, or synthetic.

Provenance helps later. If synthetic cases pass and production cases fail, the synthetic ones were too easy.

Synthetic Cases

Sometimes real traffic has gaps. You have few adversarial chats, or a new feature with no users yet.

Synthetic case = a case written by a model, to fill a gap.

They are fast to make. They are also easy to get wrong.

The model wrote ten cases. 6 of 10 had a problem:

• Unrealistic: the order does not exist, or no shopper would ever ask it.
• Leaked answer: the message already contains the answer, so the test is too easy.
• Duplicate: a copy of a case you already have. It counts the same thing twice.

Here is the checker in code. It looks for leaked answers and copies.

Check cases for leaked answers and copies

type Case = { id: string; text: string; answer: string };

const cases: Case[] = [
  { id: "S-01", text: "Any news on order 2021? It's a gift.", answer: "Friday" },
  { id: "S-02", text: "Order 2009 arrives on Wednesday, right? When does it arrive?", answer: "Wednesday" },
  { id: "S-03", text: "Where is my order 2034?", answer: "Monday" },
  { id: "S-04", text: "Where is my order 2034?", answer: "Monday" },
  { id: "S-05", text: "Can order 2017 go to 4 Park Lane?", answer: "4 Park Lane" },
];

const seen = new Set<string>();
for (const c of cases) {
  const problems: string[] = [];
  if (c.text.includes(c.answer) && !c.text.startsWith("Can")) problems.push("answer leaked");
  if (seen.has(c.text)) problems.push("duplicate");
  seen.add(c.text);
  console.log(`${c.id} ${problems.length ? problems.join(", ") : "ok"}`);
}

Expected output:

S-01 ok
S-02 answer leaked
S-03 ok
S-04 duplicate
S-05 ok

S-05 contains "4 Park Lane", but that is the request, not the answer to a question. The checker has a rule for requests. Real checkers need rules like this too.

Try this: add a case where the answer word is hidden inside another word, such as "Mondays". Does the checker flag it? Should it?

Simple lesson: never let a synthetic case in without a check.

Where It Is Used

• Building the first eval set for a new feature.
• After error analysis, to add cases for a new failure category.
• After every fixed bug, to add a regression case.
• Before a big change, such as a new model, to fill gaps with edge cases.

Common mistakes

Check Your Understanding

Exercises

What to Remember

• The eval set decides what your eval can see.
• Build a golden set from real traffic, in the real mix, covering every failure category.
• Add edge, adversarial and regression cases, not only typical ones.
• Keep splits: the eval split for the score, the regression split must always pass.
• Record provenance for every case.
• Check every synthetic case for fakes, leaked answers and copies.

Next: Code Graders, how to write the checks that score each case, in plain TypeScript.