Offline Evals in AI Evals & Observability

Lessons in this section

1. Vibes vs Evals

   A demo makes the new prompt look better. A 200-case eval shows it is worse. Learn what an eval is, why demos fool teams, and how to make a ship decision from numbers.

2. Error Analysis First

   Before you change anything, read the failing traces. Tag each failure, group the tags into a short failure taxonomy, and count them to find the few causes behind most failures.

3. Datasets

   An eval is only as good as its cases. Build a golden set from your failure taxonomy and real traffic, with typical, edge, adversarial and regression cases, and check synthetic cases before they get in.

4. Code Graders

   Most failures can be caught by a few lines of plain code. Write TypeScript scorers for citations, forbidden actions, tool calls and length, and test each scorer against known failures before you trust it.

5. LLM-as-Judge and Calibration

   Some failures need a model to grade them. Learn to write a judge rubric, label replies yourself, measure the judge's agreement with people, and catch its biases before you trust its scores.

6. Statistics for Evals

   v3 scores 1.5 points higher than v1. Is that real, or luck? Learn bootstrap intervals, paired comparisons and sample size, so you can make an honest ship or no-ship call.

7. Evals in CI + v1 Export

   A friendlier prompt merged with two approvals and no eval. Build a CI gate that runs the eval set on every prompt, model or retrieval change, blocks the regression, lets fine changes through, and fits the time budget. Then export your v1 eval suite.