Capstone: Decision Record and Dataset Kit

By Syncfusion®, Inc. · Last updated October 29, 2026

Decide all four requests with evidence, check your decision record against the course rubric, and download the full pack: record, dataset kit, method, eval plan and re-training checklist.

Capstone: Decision Record and Dataset Kit

You have all the pieces. Now put them together, the way you would for a real design review.

Four requests reached the team in lesson 1. Here is where each one ended.

Two requests never needed training. One needed a LoRA fine-tune. One needed a distilled student. Owning all four choices costs about $75,360 a year, and every dollar of it is written down.

Simple lesson: a good customization decision is one a stranger can read, check and maintain.

A Real-World Example

Think about a building permit. The builder does not just say "trust me, it is safe". They hand in drawings, calculations, the materials list and the inspection plan.

Anyone can check them, and the next builder knows exactly what was done.

Your decision record and dataset kit are the permit for your model.

Simple lesson: if it is not written down, it was not decided.

What You Will Learn

In this capstone, you will:

1. Make a decision for all four requests in the Decision lab.
2. Check your record against the course rubric.
3. Build and download the full pack: decision record, dataset kit, method note, eval plan and re-training checklist.
4. Put the whole course in order, one last time.

TL;DR

For each request: name the gap, show the cheaper rungs' scores, state the target, choose the cheapest rung that meets it, and give the yearly cost.

For anything that trains: show the cleaned data, the leak check, the curves, the before/after board with safety, and the re-training triggers.

Step 1: Decide

Open the Decision lab and choose a rung for each of the four requests. Tick the record boxes: baseline eval, general and safety eval, re-training trigger.

Step 2: Check Against the Rubric

A reviewer should be able to find all 9 items below in your record:

1. The gap type for each request.
2. The score of every cheaper rung tried, on the same eval.
3. The target, stated before the result.
4. The one-year cost of owning the choice.
5. A before/after board with general and safety slices.
6. A leak check of the training data.
7. When to re-train, and who owns it.
8. Data sign-off: consent, redaction and where the data lives.
9. The run notes: config, base model and library versions.

Here is a small checker that flags missing items in a record.

type Decision = { request: string; gap: string; rung: string; cheaperTried: boolean; target: boolean; boardWithSafety: boolean; leakChecked: boolean; retrainTrigger: boolean };

const record: Decision[] = [
  { request: "Refund policy", gap: "knowledge", rung: "rag", cheaperTried: true, target: true, boardWithSafety: false, leakChecked: false, retrainTrigger: true },
  { request: "JSON ticket", gap: "behaviour", rung: "lora", cheaperTried: true, target: true, boardWithSafety: true, leakChecked: true, retrainTrigger: false },
  { request: "Classify 2M", gap: "cost", rung: "full", cheaperTried: false, target: true, boardWithSafety: true, leakChecked: true, retrainTrigger: true },
];

const trains = (r: string) => ["lora", "dpo", "distill", "full"].includes(r);
for (const d of record) {
  const issues: string[] = [];
  if (!d.cheaperTried) issues.push("no evidence from cheaper rungs");
  if (!d.target) issues.push("no target");
  if (trains(d.rung) && !d.boardWithSafety) issues.push("no general/safety board");
  if (trains(d.rung) && !d.leakChecked) issues.push("no leak check");
  if (trains(d.rung) && !d.retrainTrigger) issues.push("no re-training trigger");
  if (d.rung === "full") issues.push("full fine-tune: say why LoRA is not enough");
  console.log(`${d.request}: ${issues.length ? issues.join("; ") : "ready for review"}`);
}

var record = new List<Decision>
{
    new("Refund policy", "knowledge", "rag", CheaperTried: true, Target: true, BoardWithSafety: false, LeakChecked: false, RetrainTrigger: true),
    new("JSON ticket", "behaviour", "lora", CheaperTried: true, Target: true, BoardWithSafety: true, LeakChecked: true, RetrainTrigger: false),
    new("Classify 2M", "cost", "full", CheaperTried: false, Target: true, BoardWithSafety: true, LeakChecked: true, RetrainTrigger: true),
};

bool Trains(string r) => new[] { "lora", "dpo", "distill", "full" }.Contains(r);
foreach (var d in record)
{
    var issues = new List<string>();
    if (!d.CheaperTried) issues.Add("no evidence from cheaper rungs");
    if (!d.Target) issues.Add("no target");
    if (Trains(d.Rung) && !d.BoardWithSafety) issues.Add("no general/safety board");
    if (Trains(d.Rung) && !d.LeakChecked) issues.Add("no leak check");
    if (Trains(d.Rung) && !d.RetrainTrigger) issues.Add("no re-training trigger");
    if (d.Rung == "full") issues.Add("full fine-tune: say why LoRA is not enough");
    Console.WriteLine($"{d.Request}: {(issues.Count > 0 ? string.Join("; ", issues) : "ready for review")}");
}

record Decision(string Request, string Gap, string Rung, bool CheaperTried, bool Target, bool BoardWithSafety, bool LeakChecked, bool RetrainTrigger);

Expected output:

Refund policy: ready for review
JSON ticket: no re-training trigger
Classify 2M: no evidence from cheaper rungs; full fine-tune: say why LoRA is not enough

The yearly cost in the recap adds up like this:

// One-year cost of owning each decision: build + upkeep + serving.
const decisions = [
  { request: "Refund policy", choice: "RAG", build: 3000, upkeep: 3600, servingMonth: 210 },
  { request: "JSON ticket", choice: "LoRA", build: 9150, upkeep: 14850, servingMonth: 260 },
  { request: "Classify 2M", choice: "Distil", build: 12150, upkeep: 14850, servingMonth: 310 },
  { request: "Be more accurate", choice: "Evals", build: 1200, upkeep: 3600, servingMonth: 300 },
];

let total = 0;
for (const d of decisions) {
  const year = d.build + d.upkeep + 12 * d.servingMonth;
  total += year;
  console.log(`${d.request.padEnd(17)} ${d.choice.padEnd(7)} $${year.toLocaleString("en-US")}`);
}
console.log(`total             $${total.toLocaleString("en-US")}`);

// One-year cost of owning each decision: build + upkeep + serving.
var decisions = new[]
{
    new { request = "Refund policy", choice = "RAG", build = 3000, upkeep = 3600, servingMonth = 210 },
    new { request = "JSON ticket", choice = "LoRA", build = 9150, upkeep = 14850, servingMonth = 260 },
    new { request = "Classify 2M", choice = "Distil", build = 12150, upkeep = 14850, servingMonth = 310 },
    new { request = "Be more accurate", choice = "Evals", build = 1200, upkeep = 3600, servingMonth = 300 },
};

var total = 0;
foreach (var d in decisions)
{
    var year = d.build + d.upkeep + 12 * d.servingMonth;
    total += year;
    Console.WriteLine($"{d.request.PadRight(17)} {d.choice.PadRight(7)} ${year:N0}");
}
Console.WriteLine($"total             ${total:N0}");

Expected output:

Refund policy     RAG     $9,120
JSON ticket       LoRA    $27,120
Classify 2M       Distil  $30,720
Be more accurate  Evals   $8,400
total             $75,360

Two of the four cost under $10,000 a year because they never train anything.

Step 3: Download the Pack

The full pack holds:

• customization-decision.md: your decisions, evidence and open questions.
• dataset-kit/: train and eval files, schema, cleaning script, dataset card, synthetic recipe.
• method.md: the method, memory estimates and starting settings.
• eval-plan.md: the before/after slices and the ship rule.
• retraining-checklist.md: the triggers and the upgrade steps.

For a real run, add the starter kit from the Export tab: put the dataset kit's train.jsonl, val.jsonl and eval.jsonl into its data/ folder, and follow its README.

Step 4: Same Pipeline, a Second Task

A fine-tuning pipeline is worth more when it is not tied to one task. Try it on a second one: deciding only the priority of a ticket.

What changes: the prompt, the expected output (one word instead of a JSON object) and the scoring rule. What stays the same: the eval tickets, the run loop, the split, the ship rule and every step of the release.

If you find yourself changing more than those three things, the pipeline has the first task baked into it. Pull that part out into config.

Common mistakes

Check Your Understanding

Exercises

What to Remember

• Name the gap, show the evidence, state the target, count the cost.
• Training needs clean data, a leak check, read curves and a full board.
• Safety regressions block release.
• Every fine-tune needs an owner, triggers and a rollback.
• The pack is reusable: replace the data, keep the process.

Next: Cheat Sheet, the whole course on one page. You have finished Fine-tuning & Model Customization.