# The Cost-Latency-Quality Triangle Course: [LLM Cost, Latency & Model Selection](https://www.devtutorials.ai/learn/llm-cost-latency-model-selection) ยท Section: Foundations Source: https://www.devtutorials.ai/learn/llm-cost-latency-model-selection/foundations/cost-latency-quality-triangle > Build the quality gate this course uses for every saving: score eval runs on cost, p95 latency and accuracy per slice, reject regressions, and pick the cheapest setup that passes, in TypeScript or C#. ## Cheaper Is Not Automatically Better You own the order-support agent in **Brightgrid Support**. It answers questions like "Where is my order A-1042?". The bill is high, and the replies feel slow. The quick fix looks obvious: use a smaller model. It is cheaper and faster. But is it still right? Every AI change moves three numbers at once: - **cost:** dollars per 1,000 requests - **latency:** how long the user waits, at p95 - **quality:** how often the answer is right, per kind of question A normal web optimisation keeps the output the same. An AI optimisation can change the answer. That is the difference. Press **Play**, or use **Next** to go one step at a time. - **Frame 1:** today's setup, with all three numbers. - **Frame 2:** four candidate changes on the same test set. - **Frame 3:** the quality gate rejects two of them. - **Frame 4:** the cheapest one that passes. **Simple lesson:** measure cost, latency and quality together, before you change anything. ### A Real-World Example A team moved its support bot to a smaller model. Cost fell by 90%. Overall accuracy fell only a little. A week later, finance found refunds paid twice. The small model got refunds wrong. Refunds were 3% of traffic, so the overall number hid it. **Simple lesson:** a wrong answer can cost more than every saving. ## What You Will Build In this lesson, you will learn: 1. how to score an eval run on cost, p95 latency and accuracy per slice 2. how to write a quality gate that rejects unsafe changes 3. how to pick the cheapest change that passes You will finish three files in the **quality** folder: - **Score** (`src/quality/score.ts`, or `Quality/Scoring.cs`): the three numbers. - **Gate** (`src/quality/gate.ts`, or `Quality/Gate.cs`): pass or reject, with reasons. - **Pick** (`src/quality/pick.ts`, or `Quality/Picker.cs`): the cheapest safe candidate. Open the **Code** tab. The harness has a **golden set**: 20 real questions with known right answers, tagged by **slice** (order status, address, refund, other). It also has five recorded eval runs: today's setup and four candidates. **Simple lesson:** a golden set is your safety net, and every later lesson uses it. ## Score the Three Corners Run the cell. Each setup gets accuracy, refund accuracy, cost per 1,000 requests and p95 latency. - **0.5b-all** is the cheapest, and gets one refund wrong. - **7b-full+judge** is the most accurate, and the slowest. - **1.5b-trimmed** is cheap, fast and as accurate as today. **p95 means 95% of requests are faster than this.** Users remember the slow ones, so latency targets use p95, not the average. **Latency Engineering** explains how to compute it. Here, the harness gives you a `percentile` helper. **Simple lesson:** one score per corner, and quality per slice. ## The Quality Gate A **quality gate** is code that says yes or no to a change. It compares a candidate with today's setup on the same golden set. It rejects the change when: - **accuracy drops** by more than a small allowance, such as 2 points - **a critical case regresses:** a refund the baseline got right is now wrong - **p95 latency** goes over the SLO A **critical slice** is a kind of question where a wrong answer costs real money or trust. Refunds are one. You choose them with your product owner. **Overall accuracy can hide a broken slice.** Check critical slices case by case. One regression there blocks the release, even if the total improved. **Simple lesson:** the gate decides, and it explains why. ## Pick the Cheapest Safe Change Now choose. Keep only the candidates that pass the gate. Pick the cheapest. On a tie, pick the faster one. Run the cell. With the real rules, the pick is **1.5b-trimmed**. With a loose gate, the 0.5B setup ships, with its refund mistake. Run `npm run quality` (or `dotnet run -- quality`). The judge is rejected for latency, 0.5B everywhere for quality. The pick is 1.5b-trimmed: 78% cheaper, same accuracy, refunds unchanged. **Try it on a real model.** Run a few golden questions on a real small model, and check each reply in code. Switch **Model** to **Real** in the lab's top bar. It runs an open model in your browser with WebLLM, and needs WebGPU. It is not graded. **Simple lesson:** the cheapest candidate that passes the gate wins. ## Put It in Your App - **Build the golden set from real traffic.** Take 50 to 200 real questions, write the right answer for each, and tag the slice. - **Run it on every change.** A new prompt, model or cache setting runs the golden set first, in CI. - **Grade with code where you can.** Check an order id, an amount or a JSON field in code. Use a model as a judge only for what code cannot check. - **Keep the eval run's cost and timing.** The same run gives you all three corners. **Start small.** Twenty good cases with slices beat a thousand untagged ones. Grow the set every time a real bug slips through. **For product managers:** you own the critical slices and the allowance. Decide which mistakes are never acceptable, and how much accuracy you would trade for cost or speed. **Simple lesson:** the golden set and the gate are the first code you write. ## Common mistakes | Mistake | Wrong idea | Correct idea | What goes wrong | |---|---|---|---| | Optimising cost alone | "Cheaper is better." | Check all three corners. | Quality drops unnoticed. | | One accuracy number | "Overall accuracy held." | Check critical slices case by case. | Refunds break silently. | | Average latency | "The mean is fine." | Use p95. | Slow requests stay hidden. | | No allowance | "Any drop blocks." | Allow a small, agreed drop. | A noisy eval blocks every change. | | Testing after release | "Users will tell us." | Gate before release, in CI. | Users find the bugs. | ## Check Your Understanding ## Exercises ## Build It in the Project The other files of this lesson's project are optional practice. Finish them to run the lesson's script on your own code. ## What to Remember - Every AI change moves cost, latency and quality. - A golden set with slices is the safety net. - The quality gate rejects accuracy drops, critical regressions and SLO breaks. - Pick the cheapest candidate that passes. - Every later lesson runs its saving through this gate. Next: **Where the Money and the Milliseconds Go**. You will meter every model call of a ticket and find the calls that make most of the bill.