# The Token Diet

Course: [LLM Cost, Latency & Model Selection](https://www.devtutorials.ai/learn/llm-cost-latency-model-selection) · Section: Eliminate Waste
Source: https://www.devtutorials.ai/learn/llm-cost-latency-model-selection/eliminate-waste/the-token-diet

> Cut the input tokens of every call without losing quality: select tools by intent and articles by score, trim chat history to a budget, and set max_tokens from real replies, in TypeScript or C#.

## Every Call Sends Too Much

Your Lesson 2 trace showed it: the agent steps read thousands of input tokens and write a few dozen. Input is the bill.

Open the prompt the agent builds for "Where is my order A-1042?":

- the system prompt
- **all six** tool schemas, even the refund and address tools
- the **whole** chat history
- **six** help articles from search, even the one scored 0.12
- the question: 40 tokens

Press **Play**, or use **Next** to go one step at a time.

- **The bar:** the prompt's tokens, coloured by section.
- **Frame 2:** only the tools and articles this question needs.
- **Frame 3:** only the recent chat turns.

**Simple lesson:** most of the prompt is not the question, and you pay for all of it on every call.

### A Real-World Example

A team trims the prompt by hand: "remove old history, fewer articles". The bill drops. Two weeks later, support sees more wrong delivery dates.

The trimmed prompt dropped the article with delivery times. Nobody ran the eval.

**Simple lesson:** a token diet is code with tests, and the eval says when you cut too far.

## What You Will Build

In this lesson, you will learn:

1. how to measure where a prompt's tokens go
2. how to select tools by intent and articles by score
3. how to trim chat history to a token budget without leaving gaps
4. how to set `max_tokens` from real replies, and detect a cut-off reply

You will finish three files in the **prompt** folder.

- **Select** (`src/prompt/select.ts`, or `Prompt/Select.cs`): tools for the intent, best articles.
- **History** (`src/prompt/history.ts`, or `Prompt/History.cs`): newest turns that fit a budget.
- **Output limit** (`src/prompt/outputLimit.ts`, or `Prompt/OutputLimit.cs`): `max_tokens` and truncation.

Open the **Code** tab. The harness has the tools, history and articles with their token counts. It also has `qualityKept`, which checks a prompt against what the eval says this question needs.

**Simple lesson:** the eval check sits next to the code that trims.

## Measure Before You Cut

Run the cell.

- **Help articles:** about half the prompt.
- **Tool schemas:** almost a quarter.
- **The question:** under 1%.

Real token counts come from the model's tokenizer. Most SDKs and local servers can count tokens before you send. Your `usage` records from Lesson 2 tell you the real total after the call.

**Simple lesson:** cut the biggest sections first.

## Select Tools and Articles

### Tools by Intent

Every tool schema in the request is input tokens, on every agent step. Most questions need two or three tools.

Give each tool a list of intents. Send the tools for this intent, plus any tool marked for every intent, such as "escalate to a human".

Keep the tools in their **original order**. In **Prompt Architecture and Caching**, a stable order is what lets the cache hit.

### Articles by Score

Search returns articles with a score. Keep the best few above a minimum score. A low-scoring article costs tokens and can confuse the model.

**Simple lesson:** send the tools and articles this question needs, not the ones you have.

## Trim the History

Long chats grow every turn. Keep the **newest** turns that fit a token budget.

Two rules keep the meaning intact:

- **No gaps.** Stop at the first turn that does not fit. Skipping it and keeping an older one leaves a hole in the conversation.
- **Keep the latest turn.** It is usually what the user just asked about.

Run the cell. Then change the budgets.

- **k=2, history 600:** 4,400 tokens, and the eval passes.
- **k=1:** fewer tokens, and the eval fails: the delivery-times article is gone.
- **history 400:** one turn too few, and the eval fails.

**The cheapest prompt is not the goal.** The goal is the cheapest prompt that still passes the eval. Run the eval on every change to the prompt builder, the same way you run unit tests.

**Simple lesson:** trim from the old end, and stop at the first turn that does not fit.

## Cap the Output

Output tokens cost more per token, and every one adds writing time. Most APIs take a `max_tokens` limit.

Set it from data, not from a guess:

- take the `completion_tokens` of real replies for this step
- take the 95th percentile, so 95% of replies fit
- add headroom, then round up

A reply that hits the limit is cut off. OpenAI-compatible servers report this as `finish_reason: "length"`. Anthropic's API reports `stop_reason: "max_tokens"`. Log it, and retry with a higher limit or a shorter prompt.

**Structured output also saves tokens.** Asking for a small JSON object instead of a paragraph usually cuts output tokens for agent steps. It also makes the reply easy to validate, which you will use in **Routing and Escalation**.

Run `npm run diet` (or `dotnet run -- diet`). Input tokens go from 10,480 to 4,400, which is 58% less, and the eval still passes.

**Simple lesson:** set output limits from real replies, and treat a cut-off as an error.

## Retrieve Less, Answer Short

Two more places hide tokens.

- **Retrieval:** a search returns the top articles, and every one goes into the prompt. Each extra article costs tokens on every call, and rarely adds the right answer.
- **Output format:** a reply in free text runs long. When code reads the reply, ask for JSON with a schema. It is shorter, and code can check it.

Run the cell. Going from top-2 to top-5 adds 1,140 tokens per call and finds the right article for 3 more questions out of 10. Check those 3 with the quality gate before you pay for them.

**Simple lesson:** choose top-k from labelled questions, and use structured output when code reads the reply.

**Try it on a real model.** Count real tokens, not estimates. Send the full prompt and the trimmed one to a real model, and compare input tokens and first-token time. Switch **Model** to **Real** in the lab's top bar. It runs an open model in your browser with WebLLM, and needs WebGPU. It is not graded.

## Put It in Your App

- **One prompt builder.** Build every prompt in one place, with the sections as data. Then you can measure, trim and test it.
- **Log section sizes.** Add the token count per section to your cost record. A section that grows is a cost leak.
- **Eval in CI.** Run the eval suite when the prompt builder changes, and fail the build when quality drops.

**For product managers:** ask for two numbers per feature: input tokens per call, and the eval score. A cut that lowers the first and keeps the second is a real saving.

**Try it with a real tokenizer.** Load a small model in the browser or in Ollama and count the tokens of your own prompts. The numbers in this lab are recorded counts from such a run.

**Simple lesson:** the prompt builder is code, so give it tests and an eval.

## Common mistakes

| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| Sending every tool on every step | "The model will ignore what it does not need." | Select tools by intent. | Thousands of tokens on every agent step. |
| Skipping a long turn in the middle | "Keep as many turns as fit." | Stop at the first turn that does not fit. | The history has a gap and the model misreads it. |
| Trimming without the eval | "Shorter is always better." | Check every cut against the eval. | Quality drops silently. |
| `max_tokens` from a guess | "512 is a safe number." | Use the p95 of real replies, plus headroom. | Cut-off replies, or no saving at all. |
| Ignoring `finish_reason` | "The reply came back, so it is complete." | Treat `length` as an error. | Users get half an answer. |

## Check Your Understanding

## Exercises

## Build It in the Project

The other files of this lesson's project are optional practice. Finish them to run the lesson's script on your own code.

## What to Remember

- Input tokens are usually most of an agent's bill.
- Measure the prompt by section, and cut the biggest sections first.
- Select tools by intent and articles by score.
- Trim history from the oldest end, with no gaps, and keep the latest turn.
- Set `max_tokens` from the p95 of real replies, and log cut-off replies.
- Every cut must pass the eval.

Next: **Prompt Architecture and Caching**. You will reorder the prompt so the part you still send on every call is billed at a fraction of the price.