Distillation for Cost and Latency

By Syncfusion®, Inc. · Last updated October 28, 2026

Train a small student model on a big teacher's labels to cut cost and latency, check options against quality, cost and p95 targets, and route hard cases back.

Distillation for Cost and Latency

Problem (c) from lesson 1: sort 2 million tickets a month into 12 categories. A big model does it well, but it costs $9,400 a month and is too slow.

Nobody needs a smarter model here. They need a cheaper, faster one that is almost as good.

Three targets must all hold: at least 92% right, under $1,000 a month, and p95 under 400 ms. The cheapest option inside all three is Student 1.5B: $310 a month and 180 ms.

Simple lesson: when quality is already good enough, the job is to keep it while cutting cost and time.

A Real-World Example

Think about a senior engineer who reviews every pull request. Their reviews are excellent, but they are slow and expensive, and there are hundreds of pull requests a week.

So they write down their decisions on a few thousand past reviews, and a junior engineer studies them. Soon the junior handles the routine reviews quickly, and passes the hard ones up.

The senior is the teacher. The junior is the student. That is distillation.

Simple lesson: a small model can learn a big model's habits for one narrow job.

What You Will Learn

In this lesson, you will learn:

1. What distillation is: a teacher model and a student model.
2. The quality, cost and latency triangle, and how to check an option against it.
3. How to label data with a teacher and what that costs.
4. How to choose the student size.
5. Why you route hard cases back to the teacher.

TL;DR

Distillation = train a small student model on answers from a big teacher model.

Label real inputs with the teacher, check a sample, train students of a few sizes, and ship the cheapest student that meets all three targets: quality, cost and p95.

The Triangle

Quality, cost and latency pull against each other. Bigger models are usually better, slower and dearer.

type Option = { name: string; score: number; costMonth: number; p95: number };
const target = { minScore: 92, maxCost: 1000, maxP95: 400 };

const options: Option[] = [
  { name: "Big model (teacher)", score: 94, costMonth: 9400, p95: 1800 },
  { name: "Student 0.5B", score: 89, costMonth: 180, p95: 120 },
  { name: "Student 1.5B", score: 93, costMonth: 310, p95: 180 },
  { name: "Student 3B", score: 94, costMonth: 520, p95: 260 },
];

const inside = (o: Option) => o.score >= target.minScore && o.costMonth <= target.maxCost && o.p95 <= target.maxP95;
for (const o of options) console.log(`${o.name.padEnd(20)} ${inside(o) ? "inside" : "outside"}`);
const best = options.filter(inside).sort((a, b) => a.costMonth - b.costMonth)[0];
console.log(`ship: ${best.name}, saves $${(options[0].costMonth - best.costMonth).toLocaleString("en-US")} a month`);

var target = new { MinScore = 92, MaxCost = 1000, MaxP95 = 400 };

List<Option> options = [
    new("Big model (teacher)", 94, 9400, 1800),
    new("Student 0.5B", 89, 180, 120),
    new("Student 1.5B", 93, 310, 180),
    new("Student 3B", 94, 520, 260),
];

bool Inside(Option o) => o.Score >= target.MinScore && o.CostMonth <= target.MaxCost && o.P95 <= target.MaxP95;
foreach (var o in options) Console.WriteLine($"{o.Name.PadRight(20)} {(Inside(o) ? "inside" : "outside")}");
var best = options.Where(Inside).OrderBy(o => o.CostMonth).First();
Console.WriteLine($"ship: {best.Name}, saves ${options[0].CostMonth - best.CostMonth:N0} a month");

record Option(string Name, int Score, int CostMonth, int P95);

Expected output:

Big model (teacher)  outside
Student 0.5B         outside
Student 1.5B         inside
Student 3B           inside
ship: Student 1.5B, saves $9,090 a month

Student 3B is also inside the triangle, but it costs more than Student 1.5B. When two options pass, take the cheaper one, and keep the other as a fallback.

The Pipeline

Labelling 50,000 tickets with the teacher costs about $235, once. The student saves about $9,000 a month, so that pays back in the first month.

Route the Hard Cases

A student is confident on routine tickets and unsure on strange ones. Many classifiers report a confidence: how sure they are.

A common pattern: if the student's confidence is low, send that ticket to the teacher. Most traffic stays cheap, and the hard few still get the big model.

Simple lesson: use the small model for the easy majority, and the big model for the hard minority.

Routing is not free. Here is what different shares cost at 2 million tickets a month:

// Route low-confidence tickets to the teacher. What does the blend cost?
const tickets = 2_000_000;
const studentPer1k = 0.155;  // dollars per 1,000 tickets
const teacherPer1k = 4.7;

for (const toTeacher of [0, 0.03, 0.1, 0.3]) {
  const cost = (tickets * (1 - toTeacher) / 1000) * studentPer1k + (tickets * toTeacher / 1000) * teacherPer1k;
  console.log(`${String(toTeacher * 100).padStart(2)}% to teacher: $${Math.round(cost)} a month`);
}

// Route low-confidence tickets to the teacher. What does the blend cost?
var tickets = 2_000_000;
var studentPer1k = 0.155;  // dollars per 1,000 tickets
var teacherPer1k = 4.7;

foreach (var toTeacher in new[] { 0, 0.03, 0.1, 0.3 })
{
    var cost = (tickets * (1 - toTeacher) / 1000) * studentPer1k + (tickets * toTeacher / 1000) * teacherPer1k;
    Console.WriteLine($"{Math.Round(toTeacher * 100),2}% to teacher: ${Math.Round(cost, MidpointRounding.AwayFromZero)} a month");
}

Expected output:

 0% to teacher: $310 a month
 3% to teacher: $583 a month
10% to teacher: $1219 a month
30% to teacher: $3037 a month

Sending 3% to the teacher stays inside the $1,000 budget. At 10% the budget is gone. Set the confidence threshold with the eval and the budget together.

Common mistakes

Check Your Understanding

Exercises

What to Remember

• Distillation: a small student learns a big teacher's answers.
• Check quality, cost and p95 together. Ship the cheapest option inside all three.
• Check the teacher's labels: the student copies its mistakes.
• Route low-confidence cases to the teacher.

Next: Serving and Maintaining a Fine-tune, what owning an adapter looks like over a year.