Observability and Production in AI Evals & Observability

Lessons in this section

1. Tracing with OTel GenAI Conventions

   A customer says Shop Helper refunded them without asking. The logs say "200 OK". Learn to record every model call, tool call and retrieval as spans with the standard gen_ai.* attributes, link chats with session ids, and keep personal data out.

2. Cost, Latency and SLOs

   The team switched to a bigger model. Quality went up, and so did the bill, by seven times. Learn to measure cost per request, p50 and p95 latency, error and refusal rates, set SLOs, and find the step that blows the budget.

3. Online Evals and Monitoring

   Quality starts to drift on day 4 and nobody notices for three days. Learn to judge a sample of live chats, read user feedback, and tune an alert that catches the drift within a day without crying wolf.

4. Debugging from Traces

   The quality alert fires on day 12. Follow it to the failing traces, compare a good trace with a bad one, find the stale search index, fix it, and add a regression case so it never slips through again.

5. Experiments for Prompts and Models

   A bigger model wins the demo. Before anyone switches, learn to version prompts, shadow-test a new model on live traffic, weigh quality against cost and latency, and roll a change out as a canary behind the CI gate.

6. Capstone: LLMOps Pack for Your Feature

   Put every part together for one feature: taxonomy, eval set, graders, calibrated judge, CI gate, tracing, dashboards and alerts. Test the pack against six production incidents and two pull requests, export it, and map a real team's answers onto it.