LLM Cost, Latency & Model Selection tutorial

Predict what an AI feature will cost and how fast it will feel before you ship it. Pick the right model per task, then cut cost and latency without losing quality, and prove every saving with an eval.

Start the course

Course contents

Foundations

  1. The Cost-Latency-Quality Triangle — Build the quality gate this course uses for every saving: score eval runs on cost, p95 latency and accuracy per slice, reject regressions, and pick the cheapest setup that passes, in TypeScript or C#.
  2. Where the Money and the Milliseconds Go — Add cost and latency logging to an AI agent in TypeScript or C#. Read usage from an OpenAI-compatible server, meter every streaming call and retry, and find the calls that make most of the bill.
  3. The Cost Equation — Write the full LLM cost equation in TypeScript or C#: calls per step, cached input, output and reasoning tokens. Forecast 10K, 100K and 1M requests a month, cost per user and per case, and find the driver that moves the bill most.
  4. Pricing an AI Feature — Turn cost records into unit economics in TypeScript or C#: margin per seat, break-even usage, the price for a target margin, and an allowance plan that makes money on every seat.
  5. How LLM Latency Works — Split an LLM call into network, queue, prefill and decode in TypeScript or C#, predict time to first token and total time before you ship, and set a latency budget per step against a first-text SLO.
  6. Measure in Production — Normalise token usage from OpenAI-style and Anthropic-style APIs, attribute LLM cost and p95 latency by feature, tenant and prompt version, and write a cost test that fails CI when a prompt change breaks a budget, in TypeScript or C#.

Eliminate Waste

  1. The Token Diet — Cut the input tokens of every call without losing quality: select tools by intent and articles by score, trim chat history to a budget, and set max_tokens from real replies, in TypeScript or C#.
  2. Prompt Architecture and Caching — Make the provider's prompt cache hit: put the stable prefix first, sort lists, price cache writes and reads with the TTL, and lint prompts in CI, in TypeScript or C#.
  3. Semantic Caching — Skip model calls for repeat questions without serving wrong answers: cosine similarity with a TTL, a policy that never caches personal or time-bound questions, and a threshold tuned on labelled replay, in TypeScript or C#.

Model Strategy

  1. Choosing a Model — Pick a model per task from eval results, p95 latency and cost: find the Pareto front, apply quality, latency and licence rules, and build a model plan with its monthly bill, in TypeScript or C#.
  2. Routing and Escalation — Send requests to a small model first, validate the reply in code, and escalate on failed checks, low confidence or critical slices. Prove the router with a report on cost, quality and silent errors, in TypeScript or C#.

Performance

  1. Batch Work — Move work nobody waits for to a batch API: decide per job, write a JSONL batch file with unique ids, and merge results back in order while collecting failures, in TypeScript or C#.
  2. Latency Engineering — Cut the time users wait: schedule model calls as a graph, measure p50, p95 and p99 against an SLO, and use streaming, a non-blocking judge and an acknowledgement, in TypeScript or C#.

Operate at Scale

  1. Budgets, Limits and Cost Alerts — Put limits around every model call: budget checks per user, feature and day, step, retry and cost caps for agent loops, and an alert on cost per request against a rolling median, in TypeScript or C#.
  2. Build vs Buy: Open-Weight, Self-Host or API — Decide when running an open model yourself starts to pay: size GPUs for the peak, compute the total cost of ownership with people and fixed costs, and find the crossover volume, in TypeScript or C#.
  3. Capstone: A Cost-and-Latency Kit for Your Feature — Take the expensive support agent, apply every lever from the course one at a time, check each against the quality gate and the latency SLO, and export a cost-and-latency kit with a ship or block decision, in TypeScript or C#.