Caching and Cost

By the Syncfusion DevTutorials team · Last updated October 20, 2026

How prompt caching reuses the stable start of a request, what to cache, what breaks it, how input, output and cached tokens are priced, and how to cut cost and waiting time.

Think about a teacher who teaches the same lesson to five classes. The first time, she writes all the rules and examples on the board. The next class doesn't get a fresh board. She keeps what stays the same and only changes the new questions at the bottom.

AI services can do the same thing. When many requests start with the same text, the service can reuse the work it already did on that text. This is called prompt caching. It makes replies faster and cheaper, but only if the shared part is written the same way, every time, at the start.

The main idea: put what stays the same first, keep it exactly the same, and send only what each request really needs.

A Real-World Example

Think about ordering at your favourite café.

• Your first visit, you explain your whole order: oat milk, no sugar, extra hot, large.
• The barista remembers it. Next time you just say, "The usual, please."
• If you change one word at the start, "The usual, but with soy milk", the barista has to rebuild the whole order from scratch.

The saving only works when the start of every order is exactly the same.

In simple words: keep the start of every request identical, so the service can reuse its work.

What You Will Learn

In this lesson, you will learn:

1. What prompt caching is, and what is reused.
2. Which parts of a context are good to cache.
3. How to design a stable prefix, and what breaks it.
4. How input, output, and cached tokens are priced.
5. How to cut cost: minimum context, summaries, fewer tools, and less retrieval.
6. How context size affects the wait for the first word.

Cache Fundamentals

Prefix Caching

A prefix is the start of something. In a request, the prefix is everything from the first token up to the point where two requests start to differ.

Prefix caching means the service stores its work on a prefix. When a new request starts with exactly the same tokens, the service reuses that work instead of doing it again.

Find how much of two requests is shared

const stable = "You are the support assistant for Brightgrid.\nNever share other customers' details.\nReply in the format: Answer, Steps, Source.\n";

const requestA = stable + "Question: Why did my export fail?";
const requestB = stable + "Question: How do I change the theme?";

function sharedPrefix(a, b) {
  let i = 0;
  while (i < a.length && a[i] === b[i]) i++;
  return i;
}

const shared = sharedPrefix(requestA, requestB);
console.log("Shared characters:", shared, "of", requestA.length);
console.log("Shared tokens: about", Math.ceil(shared / 4));

Expected output:

Shared characters: 138 of 161
Shared tokens: about 35

Only the question is different. Everything before it can be reused, which is most of the request.

KV Cache Reuse

What exactly is reused? When a model reads tokens, it builds some internal notes for each one, called keys and values. Together they are called the KV cache. Building them is the slow part of reading a long context.

If the next request starts with the same tokens, the same keys and values would be built again. KV cache reuse skips that work by keeping them from last time. That is why the prefix must match token for token: one changed token near the start changes everything after it.

Latency Reduction

Because the model doesn't need to re-read a cached prefix, the reply starts sooner. With long stable prefixes, such as a big policy document, providers report much shorter waits before the first word. The exact saving depends on the provider and the prefix length.

Cache Candidates

Good things to cache are large, used on many requests, and rarely changed.

System Instructions

The system message is the same for every request of the same app. It is the first thing in the context, so it is the easiest part to cache.

Few-Shot Examples

Few-shot examples are sample questions with good answers. They are often long, and they don't change between requests. Put them right after the system instructions.

JSON Schemas

Output schemas and tool definitions are fixed text. Keep them in the stable prefix, and keep their order the same.

Policy Documents

A long document that many questions use, such as a returns policy or an employee handbook, is a strong candidate. Sending it once and reusing it can save a lot, compared with sending it in full on every request.

Prefix Design

Stable Context

Stable context is the part that is the same for many requests. For caching, stable context must be more than "about the same". It must be identical: same words, same order, same spaces.

Things that quietly break a prefix:

• A date or time at the top, such as "Today is 10:02".
• A user's name in the system message.
• Tool definitions listed in a different order each time.
• Small edits to the instructions for each request.

A timestamp at the top breaks the shared prefix

const stable = "You are the support assistant for Brightgrid.\nNever share other customers' details.\n";

function sharedPrefix(a, b) {
  let i = 0;
  while (i < a.length && a[i] === b[i]) i++;
  return i;
}

const badA = "Time: 10:02:13\n" + stable + "Question: Why did my export fail?";
const badB = "Time: 10:02:47\n" + stable + "Question: How do I change the theme?";

const goodA = stable + "Time: 10:02:13\nQuestion: Why did my export fail?";
const goodB = stable + "Time: 10:02:47\nQuestion: How do I change the theme?";

console.log("Time at the top, shared characters:", sharedPrefix(badA, badB));
console.log("Time after the stable part, shared characters:", sharedPrefix(goodA, goodB));

Expected output:

Time at the top, shared characters: 12
Time after the stable part, shared characters: 96

With the time at the top, the two requests differ after 12 characters, so almost nothing can be reused. Move the time below the stable part, and the whole stable part is shared again.

Reusable Context

Reusable context means designing parts so that many requests can share them. Some ways to do it:

• Put everything stable first, then user data, then retrieved chunks, then the question.
• Move anything personal, such as the user's name, out of the system message.
• Sort tools and documents in a fixed order.
• Version your stable prefix. When you change it, change it on purpose, once, for everyone.

This order is the same one you already learned for recall in Lesson 2 and for stable and dynamic context in Lesson 3. Good structure helps caching, accuracy, and cost all at once.

Token Economics

Input Tokens

Input tokens are every token you send: rules, documents, history, and the question. You pay for all of them on every request.

Output Tokens

Output tokens are the tokens the model writes. They usually cost several times more than input tokens, because writing is slower work than reading.

Cached Tokens

Cached tokens are input tokens served from the cache. Many providers charge much less for them than for normal input tokens. Some also charge a little extra the first time a prefix is stored.

The same request, with and without caching

const price = { input: 3.0, cached: 0.3, output: 15.0 };

function cost(tokens) {
  const dollars = (tokens.input * price.input + tokens.cached * price.cached + tokens.output * price.output) / 1000000;
  return "$" + dollars.toFixed(4);
}

console.log("No caching:   ", cost({ input: 3000, cached: 0, output: 300 }));
console.log("With caching: ", cost({ input: 500, cached: 2500, output: 300 }));

Expected output:

No caching:    $0.0135
With caching:  $0.0067

The request is the same size. But 2,500 of its 3,000 input tokens come from the cache, so it costs about half as much.

Token Optimization

Caching makes repeated text cheaper. But the cheapest token is the one you don't send at all.

Minimum Context

The minimum context principle from Lesson 3 is the biggest saver: send the smallest context that is still enough.

State Summaries

Long chats get expensive because each turn re-sends the history. A state summary, from Lesson 7, keeps each turn at about the same size.

Selective Tools

Every tool definition is sent on every request. Show only the tools the task needs, as in Lesson 11.

Excessive Context

Excessive context is anything sent "just in case". Old chats, unrelated documents, and full API responses all add cost and make answers worse.

Excessive Retrieval

Excessive retrieval means sending too many search results. Sending the top 20 chunks when the top 4 are enough multiplies the cost, and buries the best chunk in the middle.

Top 20 chunks or top 4: the monthly bill

const tokensPerChunk = 250;
const pricePerMillion = 3;
const requestsPerDay = 20000;

function monthly(topK) {
  const tokens = topK * tokensPerChunk * requestsPerDay * 30;
  return "$" + Math.round((tokens / 1000000) * pricePerMillion);
}

console.log("Top 20 chunks:", monthly(20), "per month");
console.log("Top 4 chunks: ", monthly(4), "per month");

Expected output:

Top 20 chunks: $9000 per month
Top 4 chunks:  $1800 per month

Same app, same users. Sending a fifth of the chunks costs a fifth as much. With good ranking, the answers are often better too.

Latency Optimization

Prompt Optimization

The model must read the whole context before it writes its first word. A shorter context means a shorter wait. A cached prefix also shortens the wait, because that part is not read again.

Time To First Token

Time to first token is how long the user waits before the first word of the reply appears. For a chat app, it is the number users notice most.

A smaller context starts the reply sooner

const readSpeed = 2000;
const writeSpeed = 40;

function timing(inputTokens, outputTokens) {
  const firstToken = inputTokens / readSpeed;
  const total = firstToken + outputTokens / writeSpeed;
  return "first word after " + firstToken.toFixed(1) + " s, finished after " + total.toFixed(1) + " s";
}

console.log("8,000 input tokens:", timing(8000, 200));
console.log("1,000 input tokens:", timing(1000, 200));

Expected output:

8,000 input tokens: first word after 4.0 s, finished after 9.0 s
1,000 input tokens: first word after 0.5 s, finished after 5.5 s

The reply is the same length in both cases. But with the smaller context, the first word appears 3.5 seconds sooner.

Streaming Performance

Streaming means showing the reply word by word as it is written, instead of waiting for the whole thing. It doesn't make the model faster, but the user sees progress at once.

Streaming and a small context work together. A small context gets the first word out fast. Streaming keeps the words coming. The sandbox streams every reply, so you can see this.

Performance Impact

Cost Impact

Most savings come from three changes: smaller contexts, shorter answers, and caching. Shorter answers matter because output tokens cost the most.

Two changes, and the bill drops by more than half

const baseline = { input: 6000, output: 400 };

const steps = [
  { name: "Minimum context", input: 2500, output: 400 },
  { name: "Shorter answers", input: 2500, output: 200 },
];

const price = { input: 3, output: 15 };
const cost = (t) => (t.input * price.input + t.output * price.output) / 1000000;

console.log("Baseline:", "$" + cost(baseline).toFixed(4));
for (const s of steps) {
  const saved = 100 - Math.round((cost(s) / cost(baseline)) * 100);
  console.log(s.name + ":", "$" + cost(s).toFixed(4), "(" + saved + "% less)");
}

Expected output:

Baseline: $0.0240
Minimum context: $0.0135 (44% less)
Shorter answers: $0.0105 (56% less)

Cutting the context did most of the work. Halving the answer length saved more again, because output tokens are five times as expensive in this example.

Latency Impact

The same changes also cut waiting time:

• A smaller context gives a faster first word.
• A cached prefix gives a faster first word.
• Shorter answers finish sooner.
• Fewer tool calls mean fewer round trips.

Common mistakes

Exercises

Each exercise ends with a value on the last line. Press Check to compare it with the expected answer.

What to Remember

• Prefix caching reuses the work on an identical start of a request. The reused notes are the KV cache.
• Good cache candidates: system instructions, examples, schemas, tool definitions, and shared documents.
• Keep the stable prefix identical. Dates, names, and reordering break it.
• Output tokens usually cost the most. Cached tokens usually cost the least.
• Cut cost with minimum context, state summaries, selective tools, and fewer chunks.
• A smaller or cached context means a shorter time to first token. Streaming shows progress straight away.

Next: Context Security, how to stop injected instructions, keep data behind the right permissions, and approve risky actions.