# Distillation for Cost and Latency

Course: [Fine-tuning & Model Customization](https://www.devtutorials.ai/learn/fine-tuning) · Section: Train and Operate
Source: https://www.devtutorials.ai/learn/fine-tuning/train-and-operate/distillation

> Train a small student model on a big teacher's labels to cut cost and latency, check options against quality, cost and p95 targets, and route hard cases back.

## Distillation for Cost and Latency

Problem (c) from lesson 1: sort 2 million tickets a month into 12 categories. A big model does it well, but it costs **$9,400** a month and is too slow.

Nobody needs a smarter model here. They need a cheaper, faster one that is almost as good.

Three targets must all hold: at least **92%** right, under $1,000 a month, and p95 under 400 ms. The cheapest option inside all three is **Student 1.5B**: **$310** a month and **180 ms**.

**Simple lesson:** when quality is already good enough, the job is to keep it while cutting cost and time.

### A Real-World Example

Think about a senior engineer who reviews every pull request. Their reviews are excellent, but they are slow and expensive, and there are hundreds of pull requests a week.

So they write down their decisions on a few thousand past reviews, and a junior engineer studies them. Soon the junior handles the routine reviews quickly, and passes the hard ones up.

The senior is the **teacher**. The junior is the **student**. That is distillation.

**Simple lesson:** a small model can learn a big model's habits for one narrow job.

## What You Will Learn

In this lesson, you will learn:

1. What **distillation** is: a teacher model and a student model.
2. The **quality, cost and latency triangle**, and how to check an option against it.
3. How to **label data with a teacher** and what that costs.
4. How to choose the **student size**.
5. Why you **route hard cases** back to the teacher.

**Before you start**

- **p95 latency** = the time within which 95% of requests finish. It is the "slow but normal" case, covered in AI Evals & Observability.
- All numbers here are **simulated**. The Decision lab's problem (c) uses the same targets.

## TL;DR

**Distillation** = train a small **student** model on answers from a big **teacher** model.

Label real inputs with the teacher, check a sample, train students of a few sizes, and ship the cheapest student that meets all three targets: quality, cost and p95.

## The Triangle

Quality, cost and latency pull against each other. Bigger models are usually better, slower and dearer.

```ts runnable title="Check options against three targets"
type Option = { name: string; score: number; costMonth: number; p95: number };
const target = { minScore: 92, maxCost: 1000, maxP95: 400 };

const options: Option[] = [
  { name: "Big model (teacher)", score: 94, costMonth: 9400, p95: 1800 },
  { name: "Student 0.5B", score: 89, costMonth: 180, p95: 120 },
  { name: "Student 1.5B", score: 93, costMonth: 310, p95: 180 },
  { name: "Student 3B", score: 94, costMonth: 520, p95: 260 },
];

const inside = (o: Option) => o.score >= target.minScore && o.costMonth <= target.maxCost && o.p95 <= target.maxP95;
for (const o of options) console.log(`${o.name.padEnd(20)} ${inside(o) ? "inside" : "outside"}`);
const best = options.filter(inside).sort((a, b) => a.costMonth - b.costMonth)[0];
console.log(`ship: ${best.name}, saves $${(options[0].costMonth - best.costMonth).toLocaleString("en-US")} a month`);
```

```csharp runnable title="Check options against three targets"
var target = new { MinScore = 92, MaxCost = 1000, MaxP95 = 400 };

List<Option> options = [
    new("Big model (teacher)", 94, 9400, 1800),
    new("Student 0.5B", 89, 180, 120),
    new("Student 1.5B", 93, 310, 180),
    new("Student 3B", 94, 520, 260),
];

bool Inside(Option o) => o.Score >= target.MinScore && o.CostMonth <= target.MaxCost && o.P95 <= target.MaxP95;
foreach (var o in options) Console.WriteLine($"{o.Name.PadRight(20)} {(Inside(o) ? "inside" : "outside")}");
var best = options.Where(Inside).OrderBy(o => o.CostMonth).First();
Console.WriteLine($"ship: {best.Name}, saves ${options[0].CostMonth - best.CostMonth:N0} a month");

record Option(string Name, int Score, int CostMonth, int P95);
```

Expected output:

```text
Big model (teacher)  outside
Student 0.5B         outside
Student 1.5B         inside
Student 3B           inside
ship: Student 1.5B, saves $9,090 a month
```

Student 3B is also inside the triangle, but it costs more than Student 1.5B. When two options pass, take the cheaper one, and keep the other as a fallback.

## The Pipeline

Labelling 50,000 tickets with the teacher costs about **$235**, once. The student saves about $9,000 a month, so that pays back in the first month.

**Check the teacher before you copy it.**

The student learns the teacher's mistakes too. Check a sample of teacher labels by hand. If the teacher is wrong on one category, fix that with better prompts or human labels before you train.

## Route the Hard Cases

A student is confident on routine tickets and unsure on strange ones. Many classifiers report a **confidence**: how sure they are.

A common pattern: if the student's confidence is low, send that ticket to the teacher. Most traffic stays cheap, and the hard few still get the big model.

**Simple lesson:** use the small model for the easy majority, and the big model for the hard minority.

Routing is not free. Here is what different shares cost at 2 million tickets a month:

```ts runnable title="What routing to the teacher costs"
// Route low-confidence tickets to the teacher. What does the blend cost?
const tickets = 2_000_000;
const studentPer1k = 0.155;  // dollars per 1,000 tickets
const teacherPer1k = 4.7;

for (const toTeacher of [0, 0.03, 0.1, 0.3]) {
  const cost = (tickets * (1 - toTeacher) / 1000) * studentPer1k + (tickets * toTeacher / 1000) * teacherPer1k;
  console.log(`${String(toTeacher * 100).padStart(2)}% to teacher: $${Math.round(cost)} a month`);
}
```

```csharp runnable title="What routing to the teacher costs"
// Route low-confidence tickets to the teacher. What does the blend cost?
var tickets = 2_000_000;
var studentPer1k = 0.155;  // dollars per 1,000 tickets
var teacherPer1k = 4.7;

foreach (var toTeacher in new[] { 0, 0.03, 0.1, 0.3 })
{
    var cost = (tickets * (1 - toTeacher) / 1000) * studentPer1k + (tickets * toTeacher / 1000) * teacherPer1k;
    Console.WriteLine($"{Math.Round(toTeacher * 100),2}% to teacher: ${Math.Round(cost, MidpointRounding.AwayFromZero)} a month");
}
```

Expected output:

```text
 0% to teacher: $310 a month
 3% to teacher: $583 a month
10% to teacher: $1219 a month
30% to teacher: $3037 a month
```

Sending 3% to the teacher stays inside the $1,000 budget. At 10% the budget is gone. Set the confidence threshold with the eval and the budget together.

- **What are the three targets?** Quality, cost per month, p95. Written before any training.
- **How good is the teacher on our eval?** The student will not beat it.
- **Who checked the teacher's labels?** A sample, by hand.
- **What happens on low-confidence tickets?** There should be a route back to the teacher.
- **Are we allowed to train on the teacher's output?** Check the provider's terms.

**Do's and Don'ts**

- **Do** set quality, cost and latency targets first.
- **Do** label real inputs, in the real mix.
- **Do** check a sample of teacher labels by hand.
- **Do** try more than one student size.
- **Don't** expect the student to beat the teacher.
- **Don't** pick the biggest student "to be safe" if a smaller one passes.
- **Don't** forget a route for low-confidence cases.

## Common mistakes

| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| Only one target | "It is cheap, ship it." | Check quality, cost and p95 together. | A cheap model that is too inaccurate. |
| Unchecked teacher labels | "The big model is always right." | Check a sample by hand. | The student learns the teacher's mistakes. |
| One student size | "1.5B sounds about right." | Train a few sizes and measure. | You pay for more model than you need, or less than you need. |
| No fallback | "The student handles everything." | Send low-confidence cases to the teacher. | Rare tickets get confidently wrong answers. |
| Synthetic inputs only | "The teacher can invent the tickets too." | Use real inputs in the real mix. | The student is good at made-up tickets, not real ones. |

## Check Your Understanding

## Exercises

## What to Remember

- **Distillation:** a small **student** learns a big **teacher**'s answers.
- Check **quality, cost and p95** together. Ship the **cheapest** option inside all three.
- **Check the teacher's labels**: the student copies its mistakes.
- **Route low-confidence** cases to the teacher.

Next: **Serving and Maintaining a Fine-tune**, what owning an adapter looks like over a year.