Dataset Design

By Syncfusion®, Inc. · Last updated October 28, 2026

What one training example looks like in chat JSONL, why answers must follow one labelling guide, how to check coverage, and how to split the eval by customer.

Dataset Design

The team exports support tickets to use as training data. The file has 600 examples.

Someone suggests: "Let's just train on all of it and see."

Before you do, look at what a training example even is.

A fine-tune learns from examples like this one: an input, and the exact answer you want. Nothing else. If the answers are inconsistent, the model learns to be inconsistent.

Simple lesson: the dataset IS the specification. The model learns what the examples show, not what you meant.

A Real-World Example

Think about training a new hire with a pile of old emails as examples.

If half the emails are friendly and half are rude, the new hire learns that either is fine. If there are 200 emails about delivery and only 5 about broken items, they will be great at delivery questions and shaky on broken items.

And if you test them using the same emails they studied, they will look perfect, whether they understood or not.

Simple lesson: consistent examples, enough of every kind, and a test they have not seen.

What You Will Learn

In this lesson, you will learn:

1. What one training example looks like, in the chat JSONL format.
2. Why every answer must follow the same rules.
3. How to check coverage: enough examples of every kind of input.
4. How to keep an eval split that never touches training.
5. Roughly how many examples a LoRA fine-tune needs.

TL;DR

SFT example = one input and the exact output you want. SFT means supervised fine-tuning: learning from examples with the right answer given.

JSONL = a file with one JSON object per line. Each line is one example, usually in the chat format: system, user, assistant.

Make every answer follow the same rules. Count examples per kind of input. Keep an eval set apart, split by customer.

One Example, in the Chat Format

Most fine-tuning tools take the same shape. Each line holds a list of messages:

• system: the instruction, the same on every line.
• user: the input, as it arrives in production.
• assistant: the exact answer you want the model to learn.

Here is a small checker. It reads three lines from a JSONL file and reports problems.

type Msg = { role: string; content: string };
const SYSTEM = "You turn a customer message into one support ticket. Reply with JSON only.";

// Three lines from a JSONL file.
const lines = [
  `{"messages":[{"role":"system","content":"${SYSTEM}"},{"role":"user","content":"Cancel order 2518 please"},{"role":"assistant","content":"{\\"intent\\":\\"cancel\\",\\"order\\":\\"2518\\",\\"priority\\":\\"high\\",\\"summary\\":\\"Customer wants to cancel\\"}"}]}`,
  `{"messages":[{"role":"system","content":"${SYSTEM}"},{"role":"user","content":"Where is my order 2137"},{"role":"assistant","content":"Let me check that for you!"}]}`,
  `{"messages":[{"role":"system","content":"${SYSTEM}"},{"role":"user","content":"Refund the mug"}]}`,
];

function problems(line: string): string[] {
  const out: string[] = [];
  const msgs = (JSON.parse(line) as { messages: Msg[] }).messages;
  if (msgs[0]?.role !== "system" || msgs[0].content !== SYSTEM) out.push("system message differs");
  const answer = msgs.find((m) => m.role === "assistant");
  if (!answer) return [...out, "no assistant answer"];
  try {
    const t = JSON.parse(answer.content) as Record<string, unknown>;
    for (const f of ["intent", "order", "priority", "summary"]) if (!(f in t)) out.push(`missing ${f}`);
  } catch {
    out.push("answer is not JSON");
  }
  return out;
}

lines.forEach((l, i) => {
  const p = problems(l);
  console.log(`line ${i + 1}: ${p.length ? p.join(", ") : "ok"}`);
});

using System.Text.Json;

const string SYSTEM = "You turn a customer message into one support ticket. Reply with JSON only.";

// Three lines from a JSONL file.
var lines = new[]
{
    $$"""{"messages":[{"role":"system","content":"{{SYSTEM}}"},{"role":"user","content":"Cancel order 2518 please"},{"role":"assistant","content":"{\"intent\":\"cancel\",\"order\":\"2518\",\"priority\":\"high\",\"summary\":\"Customer wants to cancel\"}"}]}""",
    $$"""{"messages":[{"role":"system","content":"{{SYSTEM}}"},{"role":"user","content":"Where is my order 2137"},{"role":"assistant","content":"Let me check that for you!"}]}""",
    $$"""{"messages":[{"role":"system","content":"{{SYSTEM}}"},{"role":"user","content":"Refund the mug"}]}""",
};

List<string> Problems(string line)
{
    var output = new List<string>();
    var msgs = JsonSerializer.Deserialize<Line>(line)!.messages;
    if (msgs.FirstOrDefault()?.role != "system" || msgs[0].content != SYSTEM) output.Add("system message differs");
    var answer = msgs.FirstOrDefault(m => m.role == "assistant");
    if (answer == null) return [.. output, "no assistant answer"];
    try
    {
        using var t = JsonDocument.Parse(answer.content);
        foreach (var f in new[] { "intent", "order", "priority", "summary" })
            if (!t.RootElement.TryGetProperty(f, out _)) output.Add($"missing {f}");
    }
    catch (JsonException)
    {
        output.Add("answer is not JSON");
    }
    return output;
}

for (var i = 0; i < lines.Length; i++)
{
    var p = Problems(lines[i]);
    Console.WriteLine($"line {i + 1}: {(p.Count > 0 ? string.Join(", ", p) : "ok")}");
}

record Msg(string role, string content);
record Line(List<Msg> messages);

Expected output:

line 1: ok
line 2: answer is not JSON
line 3: no assistant answer

Line 2 answers in a friendly sentence, so the model would learn to write sentences. Line 3 has no answer at all, so there is nothing to learn.

Simple lesson: check every line with a script before training. Never trust a file by eye.

One Behaviour, Applied the Same Way

The answer column is your specification. Write down the rules for it, then make every example follow them:

• Every ticket has all four fields: intent, order, priority, summary.
• priority follows one rule: address changes, cancellations and damaged items are "high".
• order is "none" when there is no order number.
• The system message is identical on every line.

If two people labelled the data with different rules, the model learns both, and picks between them at random.

Coverage: Enough of Every Kind

Real traffic is uneven. Count your examples per intent before you train.

The raw export has 162 where-is-my-order tickets but only 24 account tickets. After cleaning, there are only 32 damaged-item and 16 account examples left.

A model sees the common cases most, so it handles them best. The rare intents are where it will fail.

// Count examples per intent and find the gap to a minimum.
const intents = ["refund", "where", "address", "cancel", "damaged", "account", "where", "refund", "damaged", "where"];
const counts: Record<string, number> = {};
for (const i of intents) counts[i] = (counts[i] ?? 0) + 1;

const MIN = 3;
for (const [intent, n] of Object.entries(counts).sort((a, b) => b[1] - a[1])) {
  const gap = Math.max(0, MIN - n);
  console.log(`${intent.padEnd(8)} ${n}${gap ? `  need ${gap} more` : ""}`);
}

// Count examples per intent and find the gap to a minimum.
var intents = new[] { "refund", "where", "address", "cancel", "damaged", "account", "where", "refund", "damaged", "where" };
var counts = new Dictionary<string, int>();
foreach (var i in intents) counts[i] = counts.GetValueOrDefault(i) + 1;

const int MIN = 3;
foreach (var (intent, n) in counts.OrderByDescending(kv => kv.Value))
{
    var gap = Math.Max(0, MIN - n);
    Console.WriteLine($"{intent.PadRight(8)} {n}{(gap > 0 ? $"  need {gap} more" : "")}");
}

Expected output:

where    3
refund   2  need 1 more
damaged  2  need 1 more
address  1  need 2 more
cancel   1  need 2 more
account  1  need 2 more

With a minimum of 75 per intent, the team needs 43 more damaged-item and 59 more account examples. They can collect more real ones, or generate them carefully, which is the Synthetic Data lesson.

Also include hard cases: two requests in one message, very short messages, angry messages, typos. If production has them, the training data needs them.

Simple lesson: the model will be weakest where your data is thinnest.

The Eval Split

The eval set is your exam. It must stay separate from training, from the very start.

Split = dividing your examples into groups: training, validation and eval.

Three rules keep the eval honest:

1. Write the eval split first, before you build the training set.
2. Split by customer, not by row. One customer writes the same way every time. If their tickets are on both sides, the model recognises them.
3. Make the eval as hard as real traffic. Include the hard cases and the rare intents.

How Many Examples?

There is no single number, but there are useful starting points:

• A format or style task with LoRA often works with a few hundred good examples.
• A narrow classification task may need a few hundred per class.
• Quality beats quantity. 300 clean, consistent examples usually beat 3,000 messy ones.

Start small, train, run the eval, and read the failures. Add examples where the model fails, not everywhere.

Common mistakes

Check Your Understanding

Exercises

What to Remember

• A training example is one input and the exact answer, one per line in JSONL.
• The answers are the specification. Follow one labelling guide on every line.
• Count examples per intent. The model is weakest where data is thinnest.
• Write the eval split first, split by customer, and never train on it.
• A few hundred clean, consistent examples is a good start for LoRA.

Next: From Tickets to Training Data, how to turn a real helpdesk export into clean, redacted training lines you are allowed to use.