Model Strategy in LLM Cost, Latency & Model Selection

Model Strategy is a chapter of the free LLM Cost, Latency & Model Selection tutorial with 2 lessons: Choosing a Model and Routing and Escalation. Plan on about 80 minutes. It has 8 exercises you run and check in your browser.

What each lesson covers:

• Choosing a Model: Pick a model per task from eval results, p95 latency and cost: find the Pareto front, apply quality, latency and licence rules, and build a model plan with its monthly bill, in TypeScript or C#.
• Routing and Escalation: Send requests to a small model first, validate the reply in code, and escalate on failed checks, low confidence or critical slices. Prove the router with a report on cost, quality and silent errors, in TypeScript or C#.

In this chapter you learn:

• Read an eval table with quality, p95 and cost per model

Before you start, it helps to have finished Eliminate Waste, because these lessons build on it.

When you finish, continue with Performance.

Lessons in this section

1. Choosing a Model

   Pick a model per task from eval results, p95 latency and cost: find the Pareto front, apply quality, latency and licence rules, and build a model plan with its monthly bill, in TypeScript or C#.

   • Read an eval table with quality, p95 and cost per model
   • Find the Pareto front of models for a task
   • Pick the cheapest model that meets quality, latency and licence rules
   • Build a model plan per task with its monthly cost
2. Routing and Escalation

   Send requests to a small model first, validate the reply in code, and escalate on failed checks, low confidence or critical slices. Prove the router with a report on cost, quality and silent errors, in TypeScript or C#.

   • Validate a structured model reply in code
   • Route requests: accept, or escalate on a failed check, low confidence or a critical slice
   • Report a router's cost, accuracy, escalation rate and silent errors