Eliminate Waste in LLM Cost, Latency & Model Selection

Eliminate Waste is a chapter of the free LLM Cost, Latency & Model Selection tutorial with 3 lessons: The Token Diet, Prompt Architecture and Caching and Semantic Caching. Plan on about 120 minutes. It has 12 exercises you run and check in your browser.

What each lesson covers:

  • The Token Diet: Cut the input tokens of every call without losing quality: select tools by intent and articles by score, trim chat history to a budget, and set max_tokens from real replies, in TypeScript or C#.
  • Prompt Architecture and Caching: Make the provider's prompt cache hit: put the stable prefix first, sort lists, price cache writes and reads with the TTL, and lint prompts in CI, in TypeScript or C#.
  • Semantic Caching: Skip model calls for repeat questions without serving wrong answers: cosine similarity with a TTL, a policy that never caches personal or time-bound questions, and a threshold tuned on labelled replay, in TypeScript or C#.

Before you start, it helps to have finished Foundations, because these lessons build on it.

When you finish, continue with Model Strategy.

Lessons in this section

  1. The Token Diet

    Cut the input tokens of every call without losing quality: select tools by intent and articles by score, trim chat history to a budget, and set max_tokens from real replies, in TypeScript or C#.

    • Measure where a prompt's tokens go, section by section
    • Select tools by intent and help articles by score
    • Trim chat history to a token budget without gaps
    • Set max_tokens from real replies and detect cut-off replies
  2. Prompt Architecture and Caching

    Make the provider's prompt cache hit: put the stable prefix first, sort lists, price cache writes and reads with the TTL, and lint prompts in CI, in TypeScript or C#.

    • Explain how prompt caches match an exact prefix, with a minimum size and a TTL
    • Order a prompt so its stable part is a repeatable prefix
    • Price cache writes and reads, and find when caching pays
    • Lint prompts in CI so the cache keeps hitting
  3. Semantic Caching

    Skip model calls for repeat questions without serving wrong answers: cosine similarity with a TTL, a policy that never caches personal or time-bound questions, and a threshold tuned on labelled replay, in TypeScript or C#.

    • Score questions with cosine similarity and find the nearest fresh entry
    • Decide which questions must never be cached
    • Replay labelled traffic and choose a threshold with zero wrong hits