# Capstone: Decision Record and Dataset Kit

Course: [Fine-tuning & Model Customization](https://www.devtutorials.ai/learn/fine-tuning) · Section: Train and Operate
Source: https://www.devtutorials.ai/learn/fine-tuning/train-and-operate/capstone

> Decide all four requests with evidence, check your decision record against the course rubric, and download the full pack: record, dataset kit, method, eval plan and re-training checklist.

## Capstone: Decision Record and Dataset Kit

You have all the pieces. Now put them together, the way you would for a real design review.

Four requests reached the team in lesson 1. Here is where each one ended.

Two requests never needed training. One needed a LoRA fine-tune. One needed a distilled student. Owning all four choices costs about **$75,360** a year, and every dollar of it is written down.

**Simple lesson:** a good customization decision is one a stranger can read, check and maintain.

### A Real-World Example

Think about a building permit. The builder does not just say "trust me, it is safe". They hand in drawings, calculations, the materials list and the inspection plan.

Anyone can check them, and the next builder knows exactly what was done.

Your decision record and dataset kit are the permit for your model.

**Simple lesson:** if it is not written down, it was not decided.

## What You Will Learn

In this capstone, you will:

1. Make a **decision** for all four requests in the Decision lab.
2. Check your record against the **course rubric**.
3. Build and download the **full pack**: decision record, dataset kit, method note, eval plan and re-training checklist.
4. Put the **whole course** in order, one last time.

**Before you start**

- This page uses everything from every lesson before this one.
- If you chose **An app I already have** in the course setup, answer the Decision lab for one real request from your app as well. The record works the same way.
- Lead track: the decision record is the page you review and sign.

## TL;DR

For each request: name the gap, show the cheaper rungs' scores, state the target, choose the cheapest rung that meets it, and give the yearly cost.

For anything that trains: show the cleaned data, the leak check, the curves, the before/after board with safety, and the re-training triggers.

## Step 1: Decide

Open the Decision lab and choose a rung for each of the four requests. Tick the record boxes: baseline eval, general and safety eval, re-training trigger.

## Step 2: Check Against the Rubric

A reviewer should be able to find all **9 items** below in your record:

1. The gap type for each request.
2. The score of every cheaper rung tried, on the same eval.
3. The target, stated before the result.
4. The one-year cost of owning the choice.
5. A before/after board with general and safety slices.
6. A leak check of the training data.
7. When to re-train, and who owns it.
8. Data sign-off: consent, redaction and where the data lives.
9. The run notes: config, base model and library versions.

Here is a small checker that flags missing items in a record.

```ts runnable title="Lint a decision record"
type Decision = { request: string; gap: string; rung: string; cheaperTried: boolean; target: boolean; boardWithSafety: boolean; leakChecked: boolean; retrainTrigger: boolean };

const record: Decision[] = [
  { request: "Refund policy", gap: "knowledge", rung: "rag", cheaperTried: true, target: true, boardWithSafety: false, leakChecked: false, retrainTrigger: true },
  { request: "JSON ticket", gap: "behaviour", rung: "lora", cheaperTried: true, target: true, boardWithSafety: true, leakChecked: true, retrainTrigger: false },
  { request: "Classify 2M", gap: "cost", rung: "full", cheaperTried: false, target: true, boardWithSafety: true, leakChecked: true, retrainTrigger: true },
];

const trains = (r: string) => ["lora", "dpo", "distill", "full"].includes(r);
for (const d of record) {
  const issues: string[] = [];
  if (!d.cheaperTried) issues.push("no evidence from cheaper rungs");
  if (!d.target) issues.push("no target");
  if (trains(d.rung) && !d.boardWithSafety) issues.push("no general/safety board");
  if (trains(d.rung) && !d.leakChecked) issues.push("no leak check");
  if (trains(d.rung) && !d.retrainTrigger) issues.push("no re-training trigger");
  if (d.rung === "full") issues.push("full fine-tune: say why LoRA is not enough");
  console.log(`${d.request}: ${issues.length ? issues.join("; ") : "ready for review"}`);
}
```

```csharp runnable title="Lint a decision record"
var record = new List<Decision>
{
    new("Refund policy", "knowledge", "rag", CheaperTried: true, Target: true, BoardWithSafety: false, LeakChecked: false, RetrainTrigger: true),
    new("JSON ticket", "behaviour", "lora", CheaperTried: true, Target: true, BoardWithSafety: true, LeakChecked: true, RetrainTrigger: false),
    new("Classify 2M", "cost", "full", CheaperTried: false, Target: true, BoardWithSafety: true, LeakChecked: true, RetrainTrigger: true),
};

bool Trains(string r) => new[] { "lora", "dpo", "distill", "full" }.Contains(r);
foreach (var d in record)
{
    var issues = new List<string>();
    if (!d.CheaperTried) issues.Add("no evidence from cheaper rungs");
    if (!d.Target) issues.Add("no target");
    if (Trains(d.Rung) && !d.BoardWithSafety) issues.Add("no general/safety board");
    if (Trains(d.Rung) && !d.LeakChecked) issues.Add("no leak check");
    if (Trains(d.Rung) && !d.RetrainTrigger) issues.Add("no re-training trigger");
    if (d.Rung == "full") issues.Add("full fine-tune: say why LoRA is not enough");
    Console.WriteLine($"{d.Request}: {(issues.Count > 0 ? string.Join("; ", issues) : "ready for review")}");
}

record Decision(string Request, string Gap, string Rung, bool CheaperTried, bool Target, bool BoardWithSafety, bool LeakChecked, bool RetrainTrigger);
```

Expected output:

```text
Refund policy: ready for review
JSON ticket: no re-training trigger
Classify 2M: no evidence from cheaper rungs; full fine-tune: say why LoRA is not enough
```

The yearly cost in the recap adds up like this:

```ts runnable title="One-year cost of all four decisions"
// One-year cost of owning each decision: build + upkeep + serving.
const decisions = [
  { request: "Refund policy", choice: "RAG", build: 3000, upkeep: 3600, servingMonth: 210 },
  { request: "JSON ticket", choice: "LoRA", build: 9150, upkeep: 14850, servingMonth: 260 },
  { request: "Classify 2M", choice: "Distil", build: 12150, upkeep: 14850, servingMonth: 310 },
  { request: "Be more accurate", choice: "Evals", build: 1200, upkeep: 3600, servingMonth: 300 },
];

let total = 0;
for (const d of decisions) {
  const year = d.build + d.upkeep + 12 * d.servingMonth;
  total += year;
  console.log(`${d.request.padEnd(17)} ${d.choice.padEnd(7)} $${year.toLocaleString("en-US")}`);
}
console.log(`total             $${total.toLocaleString("en-US")}`);
```

```csharp runnable title="One-year cost of all four decisions"
// One-year cost of owning each decision: build + upkeep + serving.
var decisions = new[]
{
    new { request = "Refund policy", choice = "RAG", build = 3000, upkeep = 3600, servingMonth = 210 },
    new { request = "JSON ticket", choice = "LoRA", build = 9150, upkeep = 14850, servingMonth = 260 },
    new { request = "Classify 2M", choice = "Distil", build = 12150, upkeep = 14850, servingMonth = 310 },
    new { request = "Be more accurate", choice = "Evals", build = 1200, upkeep = 3600, servingMonth = 300 },
};

var total = 0;
foreach (var d in decisions)
{
    var year = d.build + d.upkeep + 12 * d.servingMonth;
    total += year;
    Console.WriteLine($"{d.request.PadRight(17)} {d.choice.PadRight(7)} ${year:N0}");
}
Console.WriteLine($"total             ${total:N0}");
```

Expected output:

```text
Refund policy     RAG     $9,120
JSON ticket       LoRA    $27,120
Classify 2M       Distil  $30,720
Be more accurate  Evals   $8,400
total             $75,360
```

Two of the four cost under $10,000 a year because they never train anything.

## Step 3: Download the Pack

The full pack holds:

- **customization-decision.md:** your decisions, evidence and open questions.
- **dataset-kit/:** train and eval files, schema, cleaning script, dataset card, synthetic recipe.
- **method.md:** the method, memory estimates and starting settings.
- **eval-plan.md:** the before/after slices and the ship rule.
- **retraining-checklist.md:** the triggers and the upgrade steps.

For a real run, add the **starter kit** from the Export tab: put the dataset kit's `train.jsonl`, `val.jsonl` and `eval.jsonl` into its `data/` folder, and follow its README.

**Make it real.**

Swap the course's simulated export for your own, run clean.ts, and fill the eval plan with your own slices. The structure stays the same: only the data changes.

- Read the **gap** first. If it is knowledge, the answer is RAG, whatever else the document says.
- Look for the **plateau table**. No table, no fine-tune.
- Check the **one-year cost** and the **owner**.
- Read the **board**: any safety regression blocks it.
- Ask for the **rollback plan** before you approve.

**Do's and Don'ts**

- **Do** fill every rubric item for every request that trains.
- **Do** keep the record next to the code, and update it with each re-train.
- **Do** write open questions down instead of hiding them.
- **Don't** submit a record without the cheaper-rung evidence.
- **Don't** leave the owner blank.
- **Don't** treat simulated course numbers as your app's numbers.

## Step 4: Same Pipeline, a Second Task

A fine-tuning pipeline is worth more when it is not tied to one task. Try it on a second one: deciding only the priority of a ticket.

What changes: the prompt, the expected output (one word instead of a JSON object) and the scoring rule. What stays the same: the eval tickets, the run loop, the split, the ship rule and every step of the release.

If you find yourself changing more than those three things, the pipeline has the first task baked into it. Pull that part out into config.

## Common mistakes

| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| Record written after shipping | "We will document it later." | Write the record before training. | Decisions get justified backwards. |
| Missing evidence | "Everyone agreed prompting was not enough." | Show the scores per rung. | A reviewer cannot check the choice. |
| No owner | "The team owns it." | Name one person. | Nobody re-trains when the base is retired. |
| Pretty but empty | "A nice slide deck is enough." | Fill the rubric items, plainly. | Looks approved, cannot be maintained. |
| Course numbers reused | "The course said LoRA gets 96%." | Measure your own app. | Your decision rests on someone else's data. |

## Check Your Understanding

## Exercises

## What to Remember

- **Name the gap**, show the **evidence**, state the **target**, count the **cost**.
- Training needs **clean data, a leak check, read curves and a full board**.
- **Safety regressions block release.**
- Every fine-tune needs an **owner**, **triggers** and a **rollback**.
- The **pack** is reusable: replace the data, keep the process.

Next: **Cheat Sheet**, the whole course on one page. You have finished Fine-tuning & Model Customization.