# Prompt Architecture and Caching Course: [LLM Cost, Latency & Model Selection](https://www.devtutorials.ai/learn/llm-cost-latency-model-selection) · Section: Eliminate Waste Source: https://www.devtutorials.ai/learn/llm-cost-latency-model-selection/eliminate-waste/prompt-architecture-and-caching > Make the provider's prompt cache hit: put the stable prefix first, sort lists, price cache writes and reads with the TTL, and lint prompts in CI, in TypeScript or C#. ## The Cache That Never Hits After the token diet, each agent call still sends about 3,800 tokens that never change: the system prompt, the tool schemas and the policy articles. Many providers can cache a repeated prefix. Cached input is billed at a fraction of the normal price, and the model starts answering sooner. Brightgrid turned caching on. The hit rate stayed at 0%. Press **Play**, or use **Next** to go one step at a time. - **Frame 1:** the prompt today, with a timestamp at the top. - **Frame 2:** the same prompt with the stable parts first. - **Frame 3:** what the provider bills for a write, a read, and an expired entry. **Simple lesson:** a prompt cache matches an exact prefix, so one changing token at the top breaks it. ### A Real-World Example Two changes in the prompt builder killed the cache: - someone added "Current time: …" as the first line, so the model knows the date - the tool list came out of a set, in a different order on each request Each change looked harmless. Together they cost about twice as much input as needed. **Simple lesson:** prompt order is a cost decision, so test it. ## What You Will Build In this lesson, you will learn: 1. how prompt caches decide what to reuse 2. how to order a prompt so its stable part is an exact, repeatable prefix 3. how to price cache writes and reads, and when caching pays 4. how to lint a prompt in CI so the cache keeps hitting You will finish three files: - **Order** (`src/prompt/orderForCache.ts`, or `Prompt/OrderForCache.cs`): stable first, sorted items. - **Cache cost** (`src/cost/cacheCost.ts`, or `Cost/CacheCost.cs`): write and read prices. - **Lint** (`src/prompt/cacheLint.ts`, or `Prompt/CacheLint.cs`): a check for CI. Open the **Code** tab. The harness builds the prompt as a list of **segments**, and runs a fake prompt cache over 30 requests. **Simple lesson:** treat the prompt as data, so you can reorder and check it. ## How Prompt Caches Work Providers differ in the details. The main ideas are the same: - **Exact prefix.** The cache reuses the longest identical start of the prompt. A single different token ends the match. - **Minimum size.** Very short prefixes are not cached. Many providers need about 1,024 tokens or more. - **Write and read prices.** Some providers charge extra the first time a prefix is cached, then much less on each reuse. - **Time to live (TTL).** An entry expires after a few minutes without use. Then the next request writes it again. Some providers cache automatically. Others ask you to mark where the cached part ends. Self-hosted servers such as vLLM can reuse a shared prefix too. Check your provider's caching page for its rules and prices. In this lab, the cache keeps the leading run of stable segments, needs 1,024 tokens, bills a write at 125% and a read at 25% of the input price, and has a 5-minute TTL. Run the cell. The first segment is volatile, and the tool order changes each time. Zero hits. **Simple lesson:** if the first segment changes, nothing after it can be reused. ## Order the Prompt for the Cache The fix is a reorder: - **stable segments first**, in their original order - **volatile segments last**: timestamp, user profile, history, question - **sort any list** that comes from a set or a map, such as tool names The model still gets the same information. Only the order changes. **The model still sees the date.** Moving the timestamp to the end does not hide it. Put volatile facts in a short block after the stable prefix. **Simple lesson:** stable first, volatile last, lists sorted. ## Price Writes and Reads A cached call has three kinds of input: - **new input** at the normal price - **cache writes** at the write price - **cache reads** at the read price **cost** = new input × input price + written × write price + cached × read price + output × output price ### When Caching Pays After one write, each read saves money. With a write at 125% and a read at 25%, the write pays for itself after **one** read. When traffic is low, the entry expires between requests, and you pay the write premium again and again. Run the cell. Change the gap between requests. - **Gap up to 5 minutes:** 29 of 30 requests hit. - **Gap of 6 minutes:** every request writes, and the cost is higher than with no cache at all. **Low traffic can make caching cost more.** If requests for the same prefix are further apart than the TTL, you pay the write premium on every call. Measure the hit rate per feature before you count the saving. **Simple lesson:** count the write premium and the TTL in the saving. ## Lint the Prompt in CI The cache broke because of small changes in the prompt builder. A test catches them before they ship. The linter reports: - **VOLATILE_FIRST:** the prompt starts with a changing segment - **VOLATILE_BEFORE_STABLE:** a stable segment comes after a changing one - **UNSORTED_ITEMS:** a list is not in sorted order - **PREFIX_TOO_SHORT:** the stable prefix is under the minimum Run `npm run cache` (or `dotnet run -- cache`). The lint goes clean, the hit rate goes from 0 to 29 of 30, and the cost per 1,000 replies drops by more than half. **Simple lesson:** a lint rule keeps the cache hitting after the next refactor. ## Design the Prompt as Layers A cache-friendly prompt starts in the template, not in the cache settings. Write the template as **layers**: - **stable layers:** system prompt, tool schemas, policy, examples (the same for every call) - **dynamic layers:** date, user profile, history, the question (different every call) Keep every stable layer before every dynamic one. Give each layer an owner and a version, so a change to one is a reviewed change. Run the cell. The same layers, reordered, go from 0 to 3,800 cacheable tokens, and input cost per 1,000 calls drops from $0.6855 to $0.2580. **Simple lesson:** stable layers first, dynamic layers last, every time. **Try it on a real model.** See a reused prefix on real hardware. Ask a follow-up with the same prefix, then with a timestamp at the start, and compare the first-token times. Switch **Model** to **Real** in the lab's top bar. It runs an open model in your browser with WebLLM, and needs WebGPU. It is not graded. ## Put It in Your App - **Log cached tokens.** Your Lesson 2 cost record already has `cachedIn`. Chart the hit rate per feature. - **Lint every prompt builder.** Run the linter in unit tests with a sample request. - **Check the provider's rules.** Minimum size, TTL and prices differ, and some need an explicit cache marker. **Caching also cuts latency.** Cached tokens do not need to be read again, so time to first token drops. You will use this in **Latency Engineering**. **Simple lesson:** measure hits, lint the order, and know your provider's rules. ## Common mistakes | Mistake | Wrong idea | Correct idea | What goes wrong | |---|---|---|---| | Timestamp at the top | "The model should see the date first." | Put volatile facts after the stable prefix. | 0% hit rate. | | Tool list from a set or map | "The tools are the same, so the prompt is the same." | Sort lists before you build the prompt. | Different text each time, so no match. | | Ignoring the write premium | "Caching always saves." | Count writes, reads and the TTL. | Low-traffic features cost more. | | Assuming the hit rate | "We turned caching on." | Log `cachedIn` and chart it. | A refactor breaks the cache and nobody sees it. | | A prefix under the minimum | "Any repeated text is cached." | Check the provider's minimum size. | Nothing is cached at all. | ## Check Your Understanding ## Exercises ## Build It in the Project The other files of this lesson's project are optional practice. Finish them to run the lesson's script on your own code. ## What to Remember - A prompt cache reuses an exact prefix only. - Put stable segments first and volatile ones last, and sort lists. - Cached input is cheaper, writes can cost more, and entries expire. - Count the write premium and the TTL before you claim a saving. - Lint prompts in CI, and log `cachedIn` to watch the hit rate. Next: **Semantic Caching**. You will skip the model call entirely for questions that were already answered.