Prompt Architecture and Caching
By Syncfusion®, Inc. · Last updated
Make the provider's prompt cache hit: put the stable prefix first, sort lists, price cache writes and reads with the TTL, and lint prompts in CI, in TypeScript or C#.
What you will learn
- Explain how prompt caches match an exact prefix, with a minimum size and a TTL
- Order a prompt so its stable part is a repeatable prefix
- Price cache writes and reads, and find when caching pays
- Lint prompts in CI so the cache keeps hitting
The Cache That Never Hits
After the token diet, each agent call still sends about 3,800 tokens that never change: the system prompt, the tool schemas and the policy articles.
Many providers can cache a repeated prefix. Cached input is billed at a fraction of the normal price, and the model starts answering sooner.
Brightgrid turned caching on. The hit rate stayed at 0%.
Press Play, or use Next to go one step at a time.
- Frame 1: the prompt today, with a timestamp at the top.
- Frame 2: the same prompt with the stable parts first.
- Frame 3: what the provider bills for a write, a read, and an expired entry.
Simple lesson: a prompt cache matches an exact prefix, so one changing token at the top breaks it.
A Real-World Example
Two changes in the prompt builder killed the cache:
- someone added "Current time: …" as the first line, so the model knows the date
- the tool list came out of a set, in a different order on each request
Each change looked harmless. Together they cost about twice as much input as needed.
Simple lesson: prompt order is a cost decision, so test it.
What You Will Build
In this lesson, you will learn:
- how prompt caches decide what to reuse
- how to order a prompt so its stable part is an exact, repeatable prefix
- how to price cache writes and reads, and when caching pays
- how to lint a prompt in CI so the cache keeps hitting
You will finish three files:
- Order (
src/prompt/orderForCache.ts, orPrompt/OrderForCache.cs): stable first, sorted items. - Cache cost (
src/cost/cacheCost.ts, orCost/CacheCost.cs): write and read prices. - Lint (
src/prompt/cacheLint.ts, orPrompt/CacheLint.cs): a check for CI.
Open the Code tab. The harness builds the prompt as a list of segments, and runs a fake prompt cache over 30 requests.
Simple lesson: treat the prompt as data, so you can reorder and check it.
How Prompt Caches Work
Providers differ in the details. The main ideas are the same:
- Exact prefix. The cache reuses the longest identical start of the prompt. A single different token ends the match.
- Minimum size. Very short prefixes are not cached. Many providers need about 1,024 tokens or more.
- Write and read prices. Some providers charge extra the first time a prefix is cached, then much less on each reuse.
- Time to live (TTL). An entry expires after a few minutes without use. Then the next request writes it again.
Some providers cache automatically. Others ask you to mark where the cached part ends. Self-hosted servers such as vLLM can reuse a shared prefix too. Check your provider's caching page for its rules and prices.
In this lab, the cache keeps the leading run of stable segments, needs 1,024 tokens, bills a write at 125% and a read at 25% of the input price, and has a 5-minute TTL.
Run the cell. The first segment is volatile, and the tool order changes each time. Zero hits.
Simple lesson: if the first segment changes, nothing after it can be reused.
Order the Prompt for the Cache
The fix is a reorder:
- stable segments first, in their original order
- volatile segments last: timestamp, user profile, history, question
- sort any list that comes from a set or a map, such as tool names
The model still gets the same information. Only the order changes.
The model still sees the date. Moving the timestamp to the end does not hide it. Put volatile facts in a short block after the stable prefix.
Simple lesson: stable first, volatile last, lists sorted.
Price Writes and Reads
A cached call has three kinds of input:
- new input at the normal price
- cache writes at the write price
- cache reads at the read price
cost = new input × input price + written × write price + cached × read price + output × output price
When Caching Pays
After one write, each read saves money. With a write at 125% and a read at 25%, the write pays for itself after one read.
When traffic is low, the entry expires between requests, and you pay the write premium again and again.
Run the cell. Change the gap between requests.
- Gap up to 5 minutes: 29 of 30 requests hit.
- Gap of 6 minutes: every request writes, and the cost is higher than with no cache at all.
Low traffic can make caching cost more. If requests for the same prefix are further apart than the TTL, you pay the write premium on every call. Measure the hit rate per feature before you count the saving.
Simple lesson: count the write premium and the TTL in the saving.
Lint the Prompt in CI
The cache broke because of small changes in the prompt builder. A test catches them before they ship.
The linter reports:
- VOLATILE_FIRST: the prompt starts with a changing segment
- VOLATILE_BEFORE_STABLE: a stable segment comes after a changing one
- UNSORTED_ITEMS: a list is not in sorted order
- PREFIX_TOO_SHORT: the stable prefix is under the minimum
Run npm run cache (or dotnet run -- cache). The lint goes clean, the hit rate goes from 0 to 29 of 30, and the cost per 1,000 replies drops by more than half.
Simple lesson: a lint rule keeps the cache hitting after the next refactor.
Design the Prompt as Layers
A cache-friendly prompt starts in the template, not in the cache settings. Write the template as layers:
- stable layers: system prompt, tool schemas, policy, examples (the same for every call)
- dynamic layers: date, user profile, history, the question (different every call)
Keep every stable layer before every dynamic one. Give each layer an owner and a version, so a change to one is a reviewed change.
Run the cell. The same layers, reordered, go from 0 to 3,800 cacheable tokens, and input cost per 1,000 calls drops from $0.6855 to $0.2580.
Simple lesson: stable layers first, dynamic layers last, every time.
Try it on a real model. See a reused prefix on real hardware. Ask a follow-up with the same prefix, then with a timestamp at the start, and compare the first-token times. Switch Model to Real in the lab's top bar. It runs an open model in your browser with WebLLM, and needs WebGPU. It is not graded.
Put It in Your App
- Log cached tokens. Your Lesson 2 cost record already has
cachedIn. Chart the hit rate per feature. - Lint every prompt builder. Run the linter in unit tests with a sample request.
- Check the provider's rules. Minimum size, TTL and prices differ, and some need an explicit cache marker.
Caching also cuts latency. Cached tokens do not need to be read again, so time to first token drops. You will use this in Latency Engineering.
Simple lesson: measure hits, lint the order, and know your provider's rules.
Common mistakes
| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| Timestamp at the top | "The model should see the date first." | Put volatile facts after the stable prefix. | 0% hit rate. |
| Tool list from a set or map | "The tools are the same, so the prompt is the same." | Sort lists before you build the prompt. | Different text each time, so no match. |
| Ignoring the write premium | "Caching always saves." | Count writes, reads and the TTL. | Low-traffic features cost more. |
| Assuming the hit rate | "We turned caching on." | Log cachedIn and chart it. | A refactor breaks the cache and nobody sees it. |
| A prefix under the minimum | "Any repeated text is cached." | Check the provider's minimum size. | Nothing is cached at all. |
Check Your Understanding
Exercises
Build It in the Project
The other files of this lesson's project are optional practice. Finish them to run the lesson's script on your own code.
What to Remember
- A prompt cache reuses an exact prefix only.
- Put stable segments first and volatile ones last, and sort lists.
- Cached input is cheaper, writes can cost more, and entries expire.
- Count the write premium and the TTL before you claim a saving.
- Lint prompts in CI, and log
cachedInto watch the hit rate.
Next: Semantic Caching. You will skip the model call entirely for questions that were already answered.