AI Evals & Observability tutorial

Answer "is this AI feature good, and is it getting better or worse?" with numbers instead of vibes. Build evals, check your judge, gate releases and watch production.

Start the course

Course contents

Offline Evals

  1. Vibes vs Evals — A demo makes the new prompt look better. A 200-case eval shows it is worse. Learn what an eval is, why demos fool teams, and how to make a ship decision from numbers.
  2. Error Analysis First — Before you change anything, read the failing traces. Tag each failure, group the tags into a short failure taxonomy, and count them to find the few causes behind most failures.
  3. Datasets — An eval is only as good as its cases. Build a golden set from your failure taxonomy and real traffic, with typical, edge, adversarial and regression cases, and check synthetic cases before they get in.
  4. Code Graders — Most failures can be caught by a few lines of plain code. Write TypeScript scorers for citations, forbidden actions, tool calls and length, and test each scorer against known failures before you trust it.
  5. LLM-as-Judge and Calibration — Some failures need a model to grade them. Learn to write a judge rubric, label replies yourself, measure the judge's agreement with people, and catch its biases before you trust its scores.
  6. Statistics for Evals — v3 scores 1.5 points higher than v1. Is that real, or luck? Learn bootstrap intervals, paired comparisons and sample size, so you can make an honest ship or no-ship call.
  7. Evals in CI + v1 Export — A friendlier prompt merged with two approvals and no eval. Build a CI gate that runs the eval set on every prompt, model or retrieval change, blocks the regression, lets fine changes through, and fits the time budget. Then export your v1 eval suite.

Observability and Production

  1. Tracing with OTel GenAI Conventions — A customer says Shop Helper refunded them without asking. The logs say "200 OK". Learn to record every model call, tool call and retrieval as spans with the standard gen_ai.* attributes, link chats with session ids, and keep personal data out.
  2. Cost, Latency and SLOs — The team switched to a bigger model. Quality went up, and so did the bill, by seven times. Learn to measure cost per request, p50 and p95 latency, error and refusal rates, set SLOs, and find the step that blows the budget.
  3. Online Evals and Monitoring — Quality starts to drift on day 4 and nobody notices for three days. Learn to judge a sample of live chats, read user feedback, and tune an alert that catches the drift within a day without crying wolf.
  4. Debugging from Traces — The quality alert fires on day 12. Follow it to the failing traces, compare a good trace with a bad one, find the stale search index, fix it, and add a regression case so it never slips through again.
  5. Experiments for Prompts and Models — A bigger model wins the demo. Before anyone switches, learn to version prompts, shadow-test a new model on live traffic, weigh quality against cost and latency, and roll a change out as a canary behind the CI gate.
  6. Capstone: LLMOps Pack for Your Feature — Put every part together for one feature: taxonomy, eval set, graders, calibrated judge, CI gate, tracing, dashboards and alerts. Test the pack against six production incidents and two pull requests, export it, and map a real team's answers onto it.

Reference

  1. Cheat Sheet — Every idea from AI Evals & Observability on one page: the eval loop, datasets, graders, judge calibration, statistics, CI gates, tracing, cost and latency, online evals, debugging, experiments and the LLMOps pack.