Foundations in LLM Cost, Latency & Model Selection

Foundations is a chapter of the free LLM Cost, Latency & Model Selection tutorial with 6 lessons: The Cost-Latency-Quality Triangle, Where the Money and the Milliseconds Go, The Cost Equation, Pricing an AI Feature, How LLM Latency Works and Measure in Production. Plan on about 225 minutes. It has 24 exercises you run and check in your browser.

What each lesson covers:

  • The Cost-Latency-Quality Triangle: Build the quality gate this course uses for every saving: score eval runs on cost, p95 latency and accuracy per slice, reject regressions, and pick the cheapest setup that passes, in TypeScript or C#.
  • Where the Money and the Milliseconds Go: Add cost and latency logging to an AI agent in TypeScript or C#. Read usage from an OpenAI-compatible server, meter every streaming call and retry, and find the calls that make most of the bill.
  • The Cost Equation: Write the full LLM cost equation in TypeScript or C#: calls per step, cached input, output and reasoning tokens. Forecast 10K, 100K and 1M requests a month, cost per user and per case, and find the driver that moves the bill most.
  • Pricing an AI Feature: Turn cost records into unit economics in TypeScript or C#: margin per seat, break-even usage, the price for a target margin, and an allowance plan that makes money on every seat.
  • How LLM Latency Works: Split an LLM call into network, queue, prefill and decode in TypeScript or C#, predict time to first token and total time before you ship, and set a latency budget per step against a first-text SLO.
  • Measure in Production: Normalise token usage from OpenAI-style and Anthropic-style APIs, attribute LLM cost and p95 latency by feature, tenant and prompt version, and write a cost test that fails CI when a prompt change breaks a budget, in TypeScript or C#.

When you finish, continue with Eliminate Waste.

Lessons in this section

  1. The Cost-Latency-Quality Triangle

    Build the quality gate this course uses for every saving: score eval runs on cost, p95 latency and accuracy per slice, reject regressions, and pick the cheapest setup that passes, in TypeScript or C#.

    • Score an eval run on all three corners: accuracy per slice, cost per 1,000 requests and p95 latency
    • Write a quality gate that rejects accuracy drops, critical-slice regressions and SLO breaks
    • Pick the cheapest candidate that passes the gate, and report the saving
  2. Where the Money and the Milliseconds Go

    Add cost and latency logging to an AI agent in TypeScript or C#. Read usage from an OpenAI-compatible server, meter every streaming call and retry, and find the calls that make most of the bill.

    • Read prompt, completion and cached tokens from an OpenAI-compatible usage field
    • Price one call from its usage and a dated price table
    • Stream with include_usage and measure time to first token and total time
    • Meter every model call and every retry in one wrapper, with a request id
  3. The Cost Equation

    Write the full LLM cost equation in TypeScript or C#: calls per step, cached input, output and reasoning tokens. Forecast 10K, 100K and 1M requests a month, cost per user and per case, and find the driver that moves the bill most.

    • Price one request from average calls per step, cached input, output and reasoning tokens
    • Forecast monthly cost, cost per user and cost per case at 10K, 100K and 1M requests
    • Project compound monthly growth
    • Rank cost drivers with a sensitivity check
  4. Pricing an AI Feature

    Turn cost records into unit economics in TypeScript or C#: margin per seat, break-even usage, the price for a target margin, and an allowance plan that makes money on every seat.

    • Compute cost, revenue and margin per seat and for a whole plan
    • Find the break-even usage of a seat
    • Find the lowest price that reaches a target margin
    • Design and check a plan with an allowance and overage
  5. How LLM Latency Works

    Split an LLM call into network, queue, prefill and decode in TypeScript or C#, predict time to first token and total time before you ship, and set a latency budget per step against a first-text SLO.

    • Split one call's time into network, queue, prefill and decode, and compute decode speed
    • Predict time to first token and total time from prompt size, answer size and model speed
    • Set a latency budget per step and check the time to first visible text against an SLO
  6. Measure in Production

    Normalise token usage from OpenAI-style and Anthropic-style APIs, attribute LLM cost and p95 latency by feature, tenant and prompt version, and write a cost test that fails CI when a prompt change breaks a budget, in TypeScript or C#.

    • Normalise usage blocks from OpenAI-style and Anthropic-style APIs, including cache and reasoning tokens
    • Attribute cost, share and p95 latency by feature, tenant and prompt version
    • Write a CI cost test that fails when a change breaks a cost or latency budget