The Cost-Latency-Quality Triangle

By Syncfusion®, Inc. · Last updated November 2, 2026

Build the quality gate this course uses for every saving: score eval runs on cost, p95 latency and accuracy per slice, reject regressions, and pick the cheapest setup that passes, in TypeScript or C#.

What you will learn

• Score an eval run on all three corners: accuracy per slice, cost per 1,000 requests and p95 latency
• Write a quality gate that rejects accuracy drops, critical-slice regressions and SLO breaks
• Pick the cheapest candidate that passes the gate, and report the saving

Cheaper Is Not Automatically Better

You own the order-support agent in Brightgrid Support. It answers questions like "Where is my order A-1042?". The bill is high, and the replies feel slow.

The quick fix looks obvious: use a smaller model. It is cheaper and faster. But is it still right?

Every AI change moves three numbers at once:

• cost: dollars per 1,000 requests
• latency: how long the user waits, at p95
• quality: how often the answer is right, per kind of question

A normal web optimisation keeps the output the same. An AI optimisation can change the answer. That is the difference.

Press Play, or use Next to go one step at a time.

• Frame 1: today's setup, with all three numbers.
• Frame 2: four candidate changes on the same test set.
• Frame 3: the quality gate rejects two of them.
• Frame 4: the cheapest one that passes.

Simple lesson: measure cost, latency and quality together, before you change anything.

A Real-World Example

A team moved its support bot to a smaller model. Cost fell by 90%. Overall accuracy fell only a little.

A week later, finance found refunds paid twice. The small model got refunds wrong. Refunds were 3% of traffic, so the overall number hid it.

Simple lesson: a wrong answer can cost more than every saving.

What You Will Build

In this lesson, you will learn:

1. how to score an eval run on cost, p95 latency and accuracy per slice
2. how to write a quality gate that rejects unsafe changes
3. how to pick the cheapest change that passes

You will finish three files in the quality folder:

• Score (src/quality/score.ts, or Quality/Scoring.cs): the three numbers.
• Gate (src/quality/gate.ts, or Quality/Gate.cs): pass or reject, with reasons.
• Pick (src/quality/pick.ts, or Quality/Picker.cs): the cheapest safe candidate.

Open the Code tab. The harness has a golden set: 20 real questions with known right answers, tagged by slice (order status, address, refund, other). It also has five recorded eval runs: today's setup and four candidates.

Simple lesson: a golden set is your safety net, and every later lesson uses it.

Score the Three Corners

Run the cell. Each setup gets accuracy, refund accuracy, cost per 1,000 requests and p95 latency.

• 0.5b-all is the cheapest, and gets one refund wrong.
• 7b-full+judge is the most accurate, and the slowest.
• 1.5b-trimmed is cheap, fast and as accurate as today.

Simple lesson: one score per corner, and quality per slice.

The Quality Gate

A quality gate is code that says yes or no to a change. It compares a candidate with today's setup on the same golden set. It rejects the change when:

• accuracy drops by more than a small allowance, such as 2 points
• a critical case regresses: a refund the baseline got right is now wrong
• p95 latency goes over the SLO

A critical slice is a kind of question where a wrong answer costs real money or trust. Refunds are one. You choose them with your product owner.

Simple lesson: the gate decides, and it explains why.

Pick the Cheapest Safe Change

Now choose. Keep only the candidates that pass the gate. Pick the cheapest. On a tie, pick the faster one.

Run the cell. With the real rules, the pick is 1.5b-trimmed. With a loose gate, the 0.5B setup ships, with its refund mistake.

Run npm run quality (or dotnet run -- quality). The judge is rejected for latency, 0.5B everywhere for quality. The pick is 1.5b-trimmed: 78% cheaper, same accuracy, refunds unchanged.

Try it on a real model. Run a few golden questions on a real small model, and check each reply in code. Switch Model to Real in the lab's top bar. It runs an open model in your browser with WebLLM, and needs WebGPU. It is not graded.

Simple lesson: the cheapest candidate that passes the gate wins.

Put It in Your App

• Build the golden set from real traffic. Take 50 to 200 real questions, write the right answer for each, and tag the slice.
• Run it on every change. A new prompt, model or cache setting runs the golden set first, in CI.
• Grade with code where you can. Check an order id, an amount or a JSON field in code. Use a model as a judge only for what code cannot check.
• Keep the eval run's cost and timing. The same run gives you all three corners.

Simple lesson: the golden set and the gate are the first code you write.

Common mistakes

Check Your Understanding

Exercises

Build It in the Project

The other files of this lesson's project are optional practice. Finish them to run the lesson's script on your own code.

What to Remember

• Every AI change moves cost, latency and quality.
• A golden set with slices is the safety net.
• The quality gate rejects accuracy drops, critical regressions and SLO breaks.
• Pick the cheapest candidate that passes.
• Every later lesson runs its saving through this gate.

Next: Where the Money and the Milliseconds Go. You will meter every model call of a ticket and find the calls that make most of the bill.