Performance in LLM Cost, Latency & Model Selection

Performance is a chapter of the free LLM Cost, Latency & Model Selection tutorial with 2 lessons: Batch Work and Latency Engineering. Plan on about 75 minutes. It has 8 exercises you run and check in your browser.

What each lesson covers:

  • Batch Work: Move work nobody waits for to a batch API: decide per job, write a JSONL batch file with unique ids, and merge results back in order while collecting failures, in TypeScript or C#.
  • Latency Engineering: Cut the time users wait: schedule model calls as a graph, measure p50, p95 and p99 against an SLO, and use streaming, a non-blocking judge and an acknowledgement, in TypeScript or C#.

In this chapter you learn:

  • Decide per job between the real-time API and a batch API
  • Schedule model calls as a dependency graph, in a row or in parallel

Before you start, it helps to have finished Model Strategy, because these lessons build on it.

When you finish, continue with Operate at Scale.

Lessons in this section

  1. Batch Work

    Move work nobody waits for to a batch API: decide per job, write a JSONL batch file with unique ids, and merge results back in order while collecting failures, in TypeScript or C#.

    • Decide per job between the real-time API and a batch API
    • Write a JSONL batch input file with unique custom ids
    • Merge batch results back in request order and collect failures to retry
  2. Latency Engineering

    Cut the time users wait: schedule model calls as a graph, measure p50, p95 and p99 against an SLO, and use streaming, a non-blocking judge and an acknowledgement, in TypeScript or C#.

    • Schedule model calls as a dependency graph, in a row or in parallel
    • Measure p50, p95 and p99 and check them against an SLO
    • Find when the first visible text appears with streaming, a judge and an acknowledgement