# Where the Money and the Milliseconds Go Course: [LLM Cost, Latency & Model Selection](https://www.devtutorials.ai/learn/llm-cost-latency-model-selection) · Section: Foundations Source: https://www.devtutorials.ai/learn/llm-cost-latency-model-selection/foundations/where-the-money-and-milliseconds-go > Add cost and latency logging to an AI agent in TypeScript or C#. Read usage from an OpenAI-compatible server, meter every streaming call and retry, and find the calls that make most of the bill. ## Measure Before You Optimise You own the order-support agent in **Brightgrid Support**. It answers questions like "Where is my order A-1042?". Two messages arrive in the same week: - **Finance:** the AI bill is almost nine times the estimate. - **Support:** customers wait more than 13 seconds for an answer. You cannot fix either one yet. You do not know which model calls cost the money or the time. In this lesson, you will add cost and latency logging to the agent's code. Then you will read the trace your code produces and find the hotspots. Press **Play**, or use **Next** to go one step at a time. - **Each row:** one model call the agent makes for one ticket. - **The numbers:** cost per 1,000 tickets, and seconds. - **The last frame:** the three calls that make most of the bill. **Simple lesson:** one user action is usually many model calls. You need a record of every one. ### A Real-World Example The team priced the feature as one model call per ticket: the reply. The cell below runs both numbers against the fake server. The estimate prices the reply call only. The metered number runs the real agent and logs every call. Press **Run**. Then change `perDay` to your own traffic and run it again. - **estimate:** one reply call per ticket, per month. - **metered:** every call the agent really makes. - **ratio:** how far off the estimate is. **Simple lesson:** an estimate that counts one call per user action is usually wrong for agents. ## What You Will Build In this lesson, you will learn: 1. what an OpenAI-compatible server returns in `usage`, and how to price it 2. why a streaming call has no `usage` unless you ask for it 3. how to measure time to first token and total time 4. how to log one cost record per call, retries included 5. how to roll records into a report per user action: cost, wait time and hotspots You will finish three files in the course project, in TypeScript or C#. Pick the language in the top bar. - **Cost record** (`src/cost/costRecord.ts`, or `Cost/CostRecord.cs`): turns `usage` into a priced record. - **Meter** (`src/cost/meter.ts`, or `Cost/Meter.cs`): wraps every streaming call and logs a record. - **Report** (`src/cost/report.ts`, or `Cost/Report.cs`): rolls records into one report per user action. Open the **Code** tab. - **Left:** the project files. The three cost files start with `TODO` lines. - **The harness file:** the fake server, the price table and the agent. It is read-only. - **The two buttons:** run the tests, or run the agent and print the trace. **The lab server is fake, and the calls are real code.** Real servers such as Ollama, vLLM and most hosted APIs return the same OpenAI-compatible shape. In this lab, a fake client replays recorded replies on a fake clock. So your code gets the same numbers on every run. **Simple lesson:** cost tracking is normal application code. It lives next to your model client. ## Read the Usage Field Every reply from an OpenAI-compatible server carries a `usage` object. - **`prompt_tokens`** = all the input tokens, cached ones included. - **`completion_tokens`** = the output tokens. - **`prompt_tokens_details.cached_tokens`** = the part of the input the server had cached. It is often missing when nothing was cached. Run the cell. Look at the three numbers and the time. Other APIs use other names. Anthropic's API, for example, uses `input_tokens`, `output_tokens` and `cache_read_input_tokens`. Map them to one record shape in one place. ### Price One Call The price list gives dollars per 1 million tokens, with a different price for each kind of token. **cost** = new input × input price + cached input × cached price + output × output price, all divided by 1,000,000 Run the cell. Then change the step to `"agent.pick_tool"` and run it again. - **input:** almost the whole cost of this agent step. - **output:** small, even though output costs more per token. Step through all three frames. The last frame previews what caching does to the same call. You will build it in **Prompt Architecture and Caching**. Open the **Prices** tab. This is the `PRICES` table your code reads. **Simple lesson:** price each kind of token at its own price, and fail loudly when a price is missing. ## Stream, Time and Meter Every Call Most chat features stream the reply, so the user sees words early. Streaming changes what you must measure. ### Usage Arrives Last, Only If You Ask With streaming, an OpenAI-compatible server usually sends `usage` in one extra chunk at the end. It only sends it when the request has `stream_options: { include_usage: true }`. Run the cell. - **First line:** no `include_usage`, so the usage is missing. A cost log built on this would show zero. - **Second line:** the same call with `include_usage`. Now the usage arrives in the last chunk. ### Time to First Token and Total Time - **Time to first token (TTFT)** = from sending the request to the first visible text. It grows with the input, because the model reads it all first. - **Total time** = from sending the request to the last chunk. Streaming hides the writing time from the user. It does not hide TTFT, and it does not hide calls that run before the reply. Step through all three frames. - **Frame 1:** one call, split into first token and writing. - **Frame 2:** the calls before the reply add up. - **Frame 3:** the background call costs money but no waiting. ### One Wrapper for Every Call Put the timing and the logging in one function that every model call goes through. Then no call is missed. The wrapper must: - add `include_usage` to the request - note the time of the first text chunk and the last chunk - log a record for **every attempt**, because a failed attempt is billed too - carry a **request id**, so all calls for one user action can be added up later **A retry is a full extra call.** Here, the agent's "read the order" step returns broken JSON, so the agent asks again. Both attempts read about 6,000 tokens. If your wrapper logs only the successful attempt, you hide one of your most expensive calls. **Simple lesson:** one wrapper, every call, every attempt, with the request id. ## Roll Up Each User Action Single records are not the answer to "what does a ticket cost?". You need one report per user action. ### The Multipliers - **Agent loops:** one call per step, and each step resends the instructions, tools and history. - **Retries:** a broken reply is asked for again. - **Judges:** a second model checks the first one. - **Extra steps:** classifiers, rewrites and summaries. Step through all three frames. - **Left:** what the estimate priced. - **Right:** what one ticket really does. - **Frame 3:** the gap per month at three traffic levels. ### Critical Path and Hotspots - **Cost per action** = the sum of all records with the same request id. - **Critical path** = the time the user waits. Add up only the calls the user waits for. Leave out background calls such as the CRM summary. - **Hotspots** = the few costliest calls. Fix these first. Run the trace command, then open the **Trace** tab. - **Each row:** a record your meter logged. - **Red rows:** the hotspots your `summarize` found. - **At the bottom:** cost per 1,000 tickets, calls, retries and the user's wait. **Check it with your own model.** Point the same wrapper at a local server such as Ollama. Its `/v1/chat/completions` endpoint returns the same `usage` shape. Compare its numbers with the lab's before you trust a saving. **Simple lesson:** group records by request id. Cost per action and the critical path come from that group. **Try it on a real model.** Your meter logs the usage numbers a real model returns. Run the ticket on a real model, and read the record it would log. Switch **Model** to **Real** in the lab's top bar. It runs an open model in your browser with WebLLM, and needs WebGPU. It is not graded. ## Put It in Your App **Two token costs hide in agents.** Tool definitions are input tokens, sent again on every agent step. Reasoning ("thinking") models bill hidden reasoning tokens as output. Your meter must count both: **The Cost Equation** shows how. Where the records go depends on your stack. - **Logs:** one structured log line per call. Easy to add up per feature, user and day. - **Traces:** OpenTelemetry has standard attributes for model calls, such as `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens`. Add cost and attempt as your own attributes. - **Prices:** keep them in one versioned file with the date they were checked. Prices change often, so never hard-code them in the call site. **For product managers:** you do not need to write this code. Ask your team for three numbers per feature. All three come from these records: - cost per user action - the user's wait - the top 3 hotspots **Simple lesson:** log every call once, in one place, with the request id and the price date. ## Common mistakes | Mistake | Wrong idea | Correct idea | What goes wrong | |---|---|---|---| | Streaming without `include_usage` | "The server always sends usage." | Ask for it in `stream_options`. | Every streamed call is logged at $0. | | Pricing `prompt_tokens` at the full input price | "Cached tokens are separate." | Subtract `cached_tokens`, then price it at the cached price. | Cached calls look more expensive than they are. | | Logging only the successful attempt | "The retry does not count." | Log every attempt with its attempt number. | The costliest calls vanish from the report. | | No request id | "We can add calls up by time." | Pass one request id through every call of an action. | You cannot say what one ticket costs. | | Adding background calls to the wait | "Every call slows the user." | Add only calls on the critical path. | You optimise calls the user never waits for. | | A missing price becomes zero | "Unknown models are rare." | Throw, or alert, when a model has no price. | A new model looks free in every report. | ## Check Your Understanding ## Exercises ## Build It in the Project The other files of this lesson's project are optional practice. Finish them to run the lesson's script on your own code. ## What to Remember - One user action usually makes many model calls. Log every one. - `usage` gives prompt, completion and cached tokens. Cached tokens are inside `prompt_tokens`. - With streaming, ask for `include_usage`, or the usage never arrives. - Time to first token grows with the input. Writing time grows with the output. - Log every attempt, with a request id, through one wrapper. - Cost per action and the critical path come from grouping records by request id. ## Further Reading - [Ollama OpenAI compatibility](https://github.com/ollama/ollama/blob/main/docs/openai.md): run the course's open models locally with the same `usage` shape. - [OpenTelemetry semantic conventions for generative AI](https://opentelemetry.io/docs/specs/semconv/gen-ai/): standard attribute names for model calls. Next: **The Cost Equation**. You will turn one ticket's records into a cost per request, a monthly forecast and the driver that moves the bill most.