# Semantic Caching

Course: [LLM Cost, Latency & Model Selection](https://www.devtutorials.ai/learn/llm-cost-latency-model-selection) · Section: Eliminate Waste
Source: https://www.devtutorials.ai/learn/llm-cost-latency-model-selection/eliminate-waste/semantic-caching

> Skip model calls for repeat questions without serving wrong answers: cosine similarity with a TTL, a policy that never caches personal or time-bound questions, and a threshold tuned on labelled replay, in TypeScript or C#.

## The Same Question, Asked Again

Help-centre questions repeat. "How do I change my delivery address?" arrives hundreds of times a day, in slightly different words.

A **semantic cache** stores past questions with their answers. When a new question means the same thing, it returns the stored answer and skips the model call.

"Means the same thing" is measured with **embeddings**: vectors where similar meanings sit close together. **Cosine similarity** scores how close two questions are, from 0 to 1.

Press **Play**, or use **Next** to go one step at a time.

- **Frame 1:** a paraphrase scores above the threshold, so the cached answer is served.
- **Frame 2:** billing address scores close to delivery address, but the answer is different.
- **Frame 3:** two order questions are almost identical, and must never share an answer.

**Simple lesson:** a semantic cache saves calls, and a wrong hit is a silent error.

### A Real-World Example

A team set the threshold to 0.85 because "85% similar sounds safe". Model calls dropped. A week later, customers asking about **billing** addresses got instructions for **delivery** addresses.

The two questions scored 0.91. The threshold was below the near miss.

**Simple lesson:** set the threshold from labelled traffic, above the near misses.

## What You Will Build

In this lesson, you will learn:

1. how to score questions with cosine similarity, and find the nearest fresh entry
2. which questions must never be cached
3. how to replay traffic and choose the threshold with zero wrong hits

You will finish three files in the **semcache** folder:

- **Similarity** (`src/semcache/similarity.ts`, or `SemCache/Similarity.cs`): cosine and the nearest entry.
- **Policy** (`src/semcache/policy.ts`, or `SemCache/Policy.cs`): what may be cached.
- **Replay** (`src/semcache/replay.ts`, or `SemCache/Replay.cs`): calls, hits, wrong hits, and the best threshold.

Open the **Code** tab. The harness has 15 help-centre questions over 75 minutes, each with an embedding and an answer group from the eval labels.

**Simple lesson:** test the cache on labelled traffic, where you know which answers are the same.

## Similarity and Freshness

Run the cell.

- **Paraphrases:** 0.94 to 0.97, same answer.
- **The near miss:** billing vs delivery, 0.91, different answer.
- **The order questions:** almost 1.00, and different answers.

The last line shows why similarity alone is not enough. Two questions can look the same and still belong to different customers.

Cached answers also go out of date. Give each entry a **time to live** and ignore older entries.

**Real embeddings come from an embedding model.** Small open embedding models run in the browser or with Ollama, and a vector store such as Redis or pgvector does the nearest-neighbour search. In this lab, the embeddings are short vectors recorded from such a model.

**Simple lesson:** match on meaning, and only against fresh entries.

## Never Cache Personal or Time-Bound Answers

Some answers belong to one user or one moment:

- **order ids** such as A-1042
- **email addresses**, "my order", "my account"
- **time words**: today, tonight, now, current

For these, skip the cache completely: no lookup and no store. Otherwise one customer can see another customer's order, or an answer from yesterday.

**A semantic cache can leak data.** If a personal answer is stored, the next similar question gets it, from a different user. Decide what may be cached before you tune any threshold.

**Simple lesson:** the cache policy comes first, the threshold second.

## Tune the Threshold on Replay

A replay runs the traffic through the cache, in order:

- personal or time-bound: call the model, no lookup, no store
- nearest fresh entry at or above the threshold: a hit, and a **wrong hit** if its answer group differs
- otherwise: call the model and store the question

Run the cell.

- **0.80 to 0.90:** more hits, and wrong answers served.
- **0.93:** 5 hits and no wrong answers.
- **0.95:** still safe, but one fewer hit.

The best threshold is the one with the fewest model calls and **zero** wrong hits.

Run `npm run semcache` (or `dotnet run -- semcache`). The best threshold is 0.93. Model calls drop from 15 to 10, which is 33% fewer, with no wrong hits.

**Simple lesson:** pick the threshold from the replay, not from a round number.

**Try it on a real model.** Ask a real model two paraphrases, and see what one cache hit would save in time and money. Switch **Model** to **Real** in the lab's top bar. It runs an open model in your browser with WebLLM, and needs WebGPU. It is not graded.

## Put It in Your App

- **Cache key and scope.** Keep separate caches per language, per product area and per prompt version. A new prompt can change the right answer.
- **Invalidate on change.** When a help article changes, drop the cached answers built from it.
- **Log hits.** Add `cacheHit` and the similarity to the cost record, and sample hits for the eval.

**Exact caching first.** Many repeated requests are byte-for-byte the same. A normal key-value cache on the exact question is safer and cheaper. Use the semantic cache for the paraphrases that remain.

**Simple lesson:** scope the cache, expire it, and log every hit.

## Common mistakes

| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| A threshold from a round number | "0.85 sounds safe." | Tune on labelled replay. | Near misses get wrong answers. |
| Caching personal questions | "Similar questions, same answer." | Never cache order ids, accounts or emails. | One user sees another's data. |
| No TTL | "Answers do not change." | Expire entries. | Old answers after a policy change. |
| Only counting hits | "More hits is better." | Count wrong hits too. | Savings with silent errors. |
| One cache for every prompt version | "The question is the same." | Scope by prompt version. | Answers from an old prompt. |

## Check Your Understanding

## Exercises

## Build It in the Project

The other files of this lesson's project are optional practice. Finish them to run the lesson's script on your own code.

## What to Remember

- A semantic cache matches by meaning, with cosine similarity.
- Personal and time-bound questions skip the cache completely.
- Entries expire after a TTL.
- Choose the threshold on labelled replay: fewest calls, zero wrong hits.
- Scope caches by prompt version, and log every hit.

Next: **Choosing a Model**. You will pick the cheapest model that passes the eval for each task, instead of one model for everything.