Operate at Scale in LLM Cost, Latency & Model Selection

Operate at Scale is a chapter of the free LLM Cost, Latency & Model Selection tutorial with 3 lessons: Budgets, Limits and Cost Alerts, Build vs Buy: Open-Weight, Self-Host or API and Capstone: A Cost-and-Latency Kit for Your Feature. Plan on about 125 minutes. It has 12 exercises you run and check in your browser.

What each lesson covers:

  • Budgets, Limits and Cost Alerts: Put limits around every model call: budget checks per user, feature and day, step, retry and cost caps for agent loops, and an alert on cost per request against a rolling median, in TypeScript or C#.
  • Build vs Buy: Open-Weight, Self-Host or API: Decide when running an open model yourself starts to pay: size GPUs for the peak, compute the total cost of ownership with people and fixed costs, and find the crossover volume, in TypeScript or C#.
  • Capstone: A Cost-and-Latency Kit for Your Feature: Take the expensive support agent, apply every lever from the course one at a time, check each against the quality gate and the latency SLO, and export a cost-and-latency kit with a ship or block decision, in TypeScript or C#.

Before you start, it helps to have finished Performance, because these lessons build on it.

Lessons in this section

  1. Budgets, Limits and Cost Alerts

    Put limits around every model call: budget checks per user, feature and day, step, retry and cost caps for agent loops, and an alert on cost per request against a rolling median, in TypeScript or C#.

    • Check a budget before every model call, per user, per feature and per day
    • Cap an agent loop by steps, retries and cost, with a clear stop reason
    • Alert on cost per request against a rolling median
  2. Build vs Buy: Open-Weight, Self-Host or API

    Decide when running an open model yourself starts to pay: size GPUs for the peak, compute the total cost of ownership with people and fixed costs, and find the crossover volume, in TypeScript or C#.

    • Size GPUs for the peak with headroom and a minimum for failover
    • Compute the monthly total cost of ownership of each serving option
    • Find the crossover volume and test how assumptions move it
  3. Capstone: A Cost-and-Latency Kit for Your Feature

    Take the expensive support agent, apply every lever from the course one at a time, check each against the quality gate and the latency SLO, and export a cost-and-latency kit with a ship or block decision, in TypeScript or C#.

    • Apply the course's levers in order and measure cost, first-text time and quality after each one
    • Build a kit report: cost model, model per step, latency plan and quality report
    • Make a ship or block decision from quality, latency and budget, and export the kit as JSON