# Dataset Design Course: [Fine-tuning & Model Customization](https://www.devtutorials.ai/learn/fine-tuning) ยท Section: Decide and Prepare Source: https://www.devtutorials.ai/learn/fine-tuning/decide-and-prepare/dataset-design > What one training example looks like in chat JSONL, why answers must follow one labelling guide, how to check coverage, and how to split the eval by customer. ## Dataset Design The team exports support tickets to use as training data. The file has **600 examples**. Someone suggests: "Let's just train on all of it and see." Before you do, look at what a training example even is. A fine-tune learns from examples like this one: an input, and the exact answer you want. Nothing else. If the answers are inconsistent, the model learns to be inconsistent. **Simple lesson:** the dataset IS the specification. The model learns what the examples show, not what you meant. ### A Real-World Example Think about training a new hire with a pile of old emails as examples. If half the emails are friendly and half are rude, the new hire learns that either is fine. If there are 200 emails about delivery and only 5 about broken items, they will be great at delivery questions and shaky on broken items. And if you test them using the same emails they studied, they will look perfect, whether they understood or not. **Simple lesson:** consistent examples, enough of every kind, and a test they have not seen. ## What You Will Learn In this lesson, you will learn: 1. What one **training example** looks like, in the chat **JSONL** format. 2. Why every answer must follow **the same rules**. 3. How to check **coverage**: enough examples of every kind of input. 4. How to keep an **eval split** that never touches training. 5. Roughly **how many examples** a LoRA fine-tune needs. **Before you start** - This lesson uses the course's **simulated ticket export**: 600 training examples and a separate eval set of 100 examples. Some problems are planted on purpose. You will clean them in the Data Quality lesson. - Open the Fine-tune Lab's **Dataset** tab to look at every row. ## TL;DR **SFT example** = one input and the exact output you want. SFT means **supervised fine-tuning**: learning from examples with the right answer given. **JSONL** = a file with one JSON object per line. Each line is one example, usually in the chat format: system, user, assistant. Make every answer follow the same rules. Count examples per kind of input. Keep an eval set apart, split by customer. ## One Example, in the Chat Format Most fine-tuning tools take the same shape. Each line holds a list of messages: - **system:** the instruction, the same on every line. - **user:** the input, as it arrives in production. - **assistant:** the exact answer you want the model to learn. Here is a small checker. It reads three lines from a JSONL file and reports problems. ```ts runnable title="Check three training lines" type Msg = { role: string; content: string }; const SYSTEM = "You turn a customer message into one support ticket. Reply with JSON only."; // Three lines from a JSONL file. const lines = [ `{"messages":[{"role":"system","content":"${SYSTEM}"},{"role":"user","content":"Cancel order 2518 please"},{"role":"assistant","content":"{\\"intent\\":\\"cancel\\",\\"order\\":\\"2518\\",\\"priority\\":\\"high\\",\\"summary\\":\\"Customer wants to cancel\\"}"}]}`, `{"messages":[{"role":"system","content":"${SYSTEM}"},{"role":"user","content":"Where is my order 2137"},{"role":"assistant","content":"Let me check that for you!"}]}`, `{"messages":[{"role":"system","content":"${SYSTEM}"},{"role":"user","content":"Refund the mug"}]}`, ]; function problems(line: string): string[] { const out: string[] = []; const msgs = (JSON.parse(line) as { messages: Msg[] }).messages; if (msgs[0]?.role !== "system" || msgs[0].content !== SYSTEM) out.push("system message differs"); const answer = msgs.find((m) => m.role === "assistant"); if (!answer) return [...out, "no assistant answer"]; try { const t = JSON.parse(answer.content) as Record; for (const f of ["intent", "order", "priority", "summary"]) if (!(f in t)) out.push(`missing ${f}`); } catch { out.push("answer is not JSON"); } return out; } lines.forEach((l, i) => { const p = problems(l); console.log(`line ${i + 1}: ${p.length ? p.join(", ") : "ok"}`); }); ``` ```csharp runnable title="Check three training lines" using System.Text.Json; const string SYSTEM = "You turn a customer message into one support ticket. Reply with JSON only."; // Three lines from a JSONL file. var lines = new[] { $$"""{"messages":[{"role":"system","content":"{{SYSTEM}}"},{"role":"user","content":"Cancel order 2518 please"},{"role":"assistant","content":"{\"intent\":\"cancel\",\"order\":\"2518\",\"priority\":\"high\",\"summary\":\"Customer wants to cancel\"}"}]}""", $$"""{"messages":[{"role":"system","content":"{{SYSTEM}}"},{"role":"user","content":"Where is my order 2137"},{"role":"assistant","content":"Let me check that for you!"}]}""", $$"""{"messages":[{"role":"system","content":"{{SYSTEM}}"},{"role":"user","content":"Refund the mug"}]}""", }; List Problems(string line) { var output = new List(); var msgs = JsonSerializer.Deserialize(line)!.messages; if (msgs.FirstOrDefault()?.role != "system" || msgs[0].content != SYSTEM) output.Add("system message differs"); var answer = msgs.FirstOrDefault(m => m.role == "assistant"); if (answer == null) return [.. output, "no assistant answer"]; try { using var t = JsonDocument.Parse(answer.content); foreach (var f in new[] { "intent", "order", "priority", "summary" }) if (!t.RootElement.TryGetProperty(f, out _)) output.Add($"missing {f}"); } catch (JsonException) { output.Add("answer is not JSON"); } return output; } for (var i = 0; i < lines.Length; i++) { var p = Problems(lines[i]); Console.WriteLine($"line {i + 1}: {(p.Count > 0 ? string.Join(", ", p) : "ok")}"); } record Msg(string role, string content); record Line(List messages); ``` Expected output: ```text line 1: ok line 2: answer is not JSON line 3: no assistant answer ``` Line 2 answers in a friendly sentence, so the model would learn to write sentences. Line 3 has no answer at all, so there is nothing to learn. **Simple lesson:** check every line with a script before training. Never trust a file by eye. ## One Behaviour, Applied the Same Way The answer column is your specification. Write down the rules for it, then make every example follow them: - Every ticket has all four fields: **intent**, **order**, **priority**, **summary**. - **priority** follows one rule: address changes, cancellations and damaged items are "high". - **order** is "none" when there is no order number. - The system message is identical on every line. If two people labelled the data with different rules, the model learns both, and picks between them at random. **Write a labelling guide first.** One page: the fields, the allowed values, and five tricky examples with the right answer. Give it to everyone who writes or checks examples. This also becomes your eval rubric. ## Coverage: Enough of Every Kind Real traffic is uneven. Count your examples per intent before you train. The raw export has **162 where-is-my-order** tickets but only **24 account** tickets. After cleaning, there are only **32 damaged-item** and **16 account** examples left. A model sees the common cases most, so it handles them best. The rare intents are where it will fail. ```ts runnable title="Count examples per intent" // Count examples per intent and find the gap to a minimum. const intents = ["refund", "where", "address", "cancel", "damaged", "account", "where", "refund", "damaged", "where"]; const counts: Record = {}; for (const i of intents) counts[i] = (counts[i] ?? 0) + 1; const MIN = 3; for (const [intent, n] of Object.entries(counts).sort((a, b) => b[1] - a[1])) { const gap = Math.max(0, MIN - n); console.log(`${intent.padEnd(8)} ${n}${gap ? ` need ${gap} more` : ""}`); } ``` ```csharp runnable title="Count examples per intent" // Count examples per intent and find the gap to a minimum. var intents = new[] { "refund", "where", "address", "cancel", "damaged", "account", "where", "refund", "damaged", "where" }; var counts = new Dictionary(); foreach (var i in intents) counts[i] = counts.GetValueOrDefault(i) + 1; const int MIN = 3; foreach (var (intent, n) in counts.OrderByDescending(kv => kv.Value)) { var gap = Math.Max(0, MIN - n); Console.WriteLine($"{intent.PadRight(8)} {n}{(gap > 0 ? $" need {gap} more" : "")}"); } ``` Expected output: ```text where 3 refund 2 need 1 more damaged 2 need 1 more address 1 need 2 more cancel 1 need 2 more account 1 need 2 more ``` With a minimum of 75 per intent, the team needs **43 more damaged-item** and **59 more account** examples. They can collect more real ones, or generate them carefully, which is the Synthetic Data lesson. Also include **hard cases**: two requests in one message, very short messages, angry messages, typos. If production has them, the training data needs them. **Simple lesson:** the model will be weakest where your data is thinnest. ## The Eval Split The eval set is your exam. It must stay separate from training, from the very start. **Split** = dividing your examples into groups: training, validation and eval. Three rules keep the eval honest: 1. **Write the eval split first**, before you build the training set. 2. **Split by customer**, not by row. One customer writes the same way every time. If their tickets are on both sides, the model recognises them. 3. **Make the eval as hard as real traffic.** Include the hard cases and the rare intents. ## How Many Examples? There is no single number, but there are useful starting points: - **A format or style task** with LoRA often works with a few hundred good examples. - **A narrow classification task** may need a few hundred per class. - **Quality beats quantity.** 300 clean, consistent examples usually beat 3,000 messy ones. Start small, train, run the eval, and read the failures. Add examples where the model fails, not everywhere. - **Is there a labelling guide**, and did everyone follow it? - **Coverage table:** examples per intent, and which ones are under the minimum. - **Eval split:** written first, split by customer, never used for training. - **Where the data came from**, and whether you are allowed to train on it (privacy, customer contracts). **Do's and Don'ts** - **Do** use the chat format with the same system message on every line. - **Do** write a labelling guide and follow it on every example. - **Do** count examples per intent and fix thin ones. - **Do** split the eval by customer, before training. - **Don't** mix answer styles in one dataset. - **Don't** pick eval examples after seeing what the model gets right. - **Don't** train on data you are not allowed to use. ## Common mistakes | Mistake | Wrong idea | Correct idea | What goes wrong | |---|---|---|---| | Training on the raw export | "More data is always better." | Check, clean and count the data first. | The model learns the export's mistakes and gaps. | | Mixed answer styles | "The model will work out what we meant." | One labelling guide for every example. | The model switches between styles at random. | | No coverage check | "The export covers everything customers ask." | Count examples per intent against a minimum. | Rare intents are handled badly in production. | | Splitting by row | "A random split is fair." | Split by customer, so no customer is on both sides. | The eval looks better than real life. | | Writing the eval last | "We will test it on whatever is left." | Write the eval split first and lock it. | The eval quietly drifts towards what the model can do. | ## Check Your Understanding ## Exercises ## What to Remember - A training example is **one input and the exact answer**, one per line in **JSONL**. - The answers are the **specification**. Follow one labelling guide on every line. - **Count examples per intent.** The model is weakest where data is thinnest. - Write the **eval split first**, split **by customer**, and never train on it. - A few hundred **clean, consistent** examples is a good start for LoRA. Next: **From Tickets to Training Data**, how to turn a real helpdesk export into clean, redacted training lines you are allowed to use.