# How LLM Latency Works Course: [LLM Cost, Latency & Model Selection](https://www.devtutorials.ai/learn/llm-cost-latency-model-selection) · Section: Foundations Source: https://www.devtutorials.ai/learn/llm-cost-latency-model-selection/foundations/how-llm-latency-works > Split an LLM call into network, queue, prefill and decode in TypeScript or C#, predict time to first token and total time before you ship, and set a latency budget per step against a first-text SLO. ## Why Does the Answer Take Ten Seconds? Users say the support agent is slow. Your logs say one ticket takes almost 11 seconds. Where do those seconds go? "The model is slow" is not an answer you can fix. You need the parts. Press **Play**, or use **Next** to go one step at a time. - **Frame 1:** one call, split into network, prefill and decode. - **Frame 2:** four calls in a row. - **Frame 3:** each step against its budget. **Simple lesson:** every call is network, queue, prefill and decode, and each part has a different fix. ### A Real-World Example A team switched to a model with faster "tokens per second". Users still waited as long. Their prompt was 20,000 tokens long, and the new model was slow at reading it. They had measured decode speed only. The wait was in prefill. **Simple lesson:** measure the parts before you buy speed. ## What You Will Build In this lesson, you will learn: 1. the four parts of a call: network, queue, prefill and decode 2. how to predict a call's time before you ship it 3. how to give each step a latency budget You will finish three files in the **timing** folder: - **Breakdown** (`src/timing/breakdown.ts`, or `Timing/Breakdown.cs`): split one measured call. - **Predict** (`src/timing/predict.ts`, or `Timing/Predict.cs`): time from tokens and model speed. - **Budget** (`src/timing/budget.ts`, or `Timing/Budget.cs`): each step against its budget. Open the **Code** tab. The harness has the measured timings of the four calls the user waits for, and each model's speeds from the course table. **Simple lesson:** latency work starts with timestamps, not guesses. ## The Four Parts of a Call - **network:** the round trip from your server to the provider - **queue:** time waiting for a free GPU at the provider, high at busy hours - **prefill:** the model reads the whole prompt, and ends at the **first token** (time to first token, TTFT) - **decode:** the model writes the answer, one token at a time Decode speed is tokens per second. The first token ends prefill, so decode writes the other tokens. Run the cell. The agent steps spend most time in prefill: long prompts. The reply spends most time in decode: a long answer. **Some providers report queue time, others do not.** When they do not, compare your measured TTFT with a quiet hour. In this lab, the queue time is in the record. **Simple lesson:** prefill grows with the prompt, decode grows with the answer. ## Predict Before You Ship With each model's speeds, you can predict a call: - **TTFT** = network + queue + start-up time + input tokens ÷ prefill speed - **total** = TTFT + output tokens ÷ decode speed Run the cell. A 20,000-token prompt adds seconds before the first word. Capping the answer at 80 tokens cuts the total, and does not change the first token. **Predictions are a plan, not a promise.** Real speeds change with load and hardware. Check predictions against your records every week, and update the speed table. **Simple lesson:** prompt size sets the first token, answer size sets the end. ## A Budget per Step The SLO says: **the user sees text within 4 seconds**. The steps run one after another, so their waits add up. - a step that is **not streamed** makes the user wait until it finishes - the **streamed** reply makes the user wait only until its first token Give each step a budget that adds up to the SLO. A step over its budget is the one to fix. Run `npm run latency-budget` (or `dotnet run -- latency-budget`). The first text arrives at 7,060 ms, and the whole answer takes 10,893 ms. Classify and both agent steps are over budget. Later lessons fix each one: a smaller model, a shorter prompt, a cached prefix and parallel calls. **Try it on a real model.** Stream a reply in your browser and see your own first-token time and decode speed. Switch **Model** to **Real** in the lab's top bar. It runs an open model in your browser with WebLLM, and needs WebGPU. It is not graded. **Simple lesson:** budget each step, and fix the ones over budget. ## Put It in Your App - **Log four timestamps.** Request sent, first byte, first token, done. With them, your meter can split every call. - **Record queue headers.** Some providers return queue or processing time in a response header. Log it when it is there. - **Keep the speed table current.** Update prefill and decode speeds from a week of records, per model. **Users notice the first word most.** A 10-second answer that starts in 1 second feels faster than a 6-second answer that starts at 6. **Latency Engineering** builds on this. **Simple lesson:** timestamps in, budgets out. **For product managers:** set the latency target as time to first visible text, at p95. Then ask engineering for a budget per step, so slow steps have an owner. ## Common mistakes | Mistake | Wrong idea | Correct idea | What goes wrong | |---|---|---|---| | One latency number | "The call took 3 s." | Split network, queue, prefill, decode. | You fix the wrong part. | | Only tokens per second | "Faster decode fixes it." | Check prefill on long prompts. | The first token stays slow. | | Ignoring the queue | "The model is slow." | Measure queue at busy hours. | You change models for nothing. | | No budget per step | "We need under 4 s." | Split the SLO by step. | Nobody owns the slow step. | | Counting a streamed reply fully | "The user waits for all of it." | Count until its first token. | You over-optimise the reply. | ## Check Your Understanding ## Exercises ## Build It in the Project The other files of this lesson's project are optional practice. Finish them to run the lesson's script on your own code. ## What to Remember - Every call is network, queue, prefill and decode. - Long prompts delay the first token, long answers delay the end. - Predict calls from tokens and model speeds before you ship. - Split the SLO into a budget per step. - A streamed step counts until its first token. Next: **Measure in Production**. You will tag every call, attribute cost by feature and fail CI when a budget breaks.