# Capstone: LLMOps Pack for Your Feature

Course: [AI Evals & Observability](https://www.devtutorials.ai/learn/evals-observability) · Section: Observability and Production
Source: https://www.devtutorials.ai/learn/evals-observability/production/capstone

> Put every part together for one feature: taxonomy, eval set, graders, calibrated judge, CI gate, tracing, dashboards and alerts. Test the pack against six production incidents and two pull requests, export it, and map a real team's answers onto it.

## Capstone: LLMOps Pack for Your Feature

Two teams ship the same feature on the same day.

Team A has a dashboard full of charts, but no eval set and no alert owners. Team B has the full pack from this course.

Over the next 14 days, six things go wrong. Team A hears about most of them from customers. Team B gets an alert for each one, with a runbook link, and knows what to do.

This lesson builds Team B's pack. First, see what is in it.

**LLMOps pack** = all the files that let you measure an AI feature, block bad changes, and watch it in production.

**Simple lesson:** the pack is the eval loop, written down as files.

### A Real-World Example

Think about a pilot before take-off.

A pilot does not trust their memory. They run a checklist. Fuel, flaps, instruments, radio. Each item was added because something once went wrong without it.

Your pack is that checklist for an AI feature. Each part is there because a team once shipped without it.

**Simple lesson:** a checklist turns lessons learned into habits.

## What You Will Learn

In this lesson, you will:

1. Put the nine parts of the pack together for Shop Helper.
2. Test the pack against six production incidents and two pull requests.
3. Decide **ship**, **block**, **roll back** or **investigate** for each.
4. Export the pack and pass the pack check.
5. Turn another team's answers into a pack for their feature.

## TL;DR

The pack has three groups:

- **Measure:** taxonomy, eval set, code graders.
- **Gate:** calibrated judge, ship rule with intervals, CI gate.
- **Watch:** tracing, dashboards with targets, online evals and alerts.

Test it the way production will. Here, the pack catches **6 of 6** seeded incidents with **0 false alerts**, and exports as **10 files**.

## Cheat Sheet

| Part | File | Lesson |
|---|---|---|
| Failure taxonomy | evals/taxonomy.md | Error Analysis First |
| Eval set with splits | evals/dataset.jsonl | Datasets |
| Code graders | evals/graders.test.ts | Code Graders |
| Calibrated judge | evals/judge-rubric.md | LLM-as-Judge and Calibration |
| CI gate | .github/workflows/evals.yml | Evals in CI |
| Tracing | otel/instrument.ts, otel/Telemetry.cs | Tracing with OTel GenAI |
| Exporter | otel/exporter.*.env | Tracing with OTel GenAI |
| Alerts | monitoring/alerts.yaml | Online Evals and Monitoring |
| Checklist | README.md | This lesson |

## Measure, Gate, Watch

Every part answers one question.

- The **taxonomy** answers: what goes wrong?
- The **eval set** answers: which cases show it?
- The **graders and judge** answer: did this reply pass?
- The **ship rule** answers: is a change really better, or is it noise?
- The **CI gate** answers: may this pull request ship?
- **Tracing** answers: what exactly happened in this chat?
- **Dashboards and alerts** answer: is it still good today?

If a part is missing, its question goes unanswered. That is where incidents hide.

**Simple lesson:** one part per question, and no question left without a part.

## Six Incidents and Two Pull Requests

The lab replays 14 days of Shop Helper traffic with six seeded incidents. Two pull requests also arrive.

Catching an incident is half the job. The other half is the right decision. There are four:

- **Ship:** a pull request passed the gate.
- **Block:** a pull request failed the gate. It never reaches users.
- **Roll back:** our own change caused it. Undo the change first, then find out why.
- **Investigate and fix:** something outside our code changed. There is nothing of ours to undo, so follow the debugging loop.

**Simple lesson:** roll back what you changed. Investigate what you did not.

## Export the Pack

The Export tab builds the whole pack as a zip. The pack check reads the files and flags problems before you ship them.

The backend tab changes only one file: the exporter config. Everything else is plain TypeScript, YAML and OpenTelemetry. You can move to another platform without rewriting your evals.

Here is the same idea in a few lines of code: a pack is ready only when every part is there.

```ts runnable title="Is the pack ready?"
// A pack manifest for a feature. Each part says which file holds it.
type Part = { name: string; file: string; done: boolean };

const pack: Part[] = [
  { name: "Failure taxonomy", file: "evals/taxonomy.md", done: true },
  { name: "Eval set", file: "evals/dataset.jsonl", done: true },
  { name: "Code graders", file: "evals/graders.test.ts", done: true },
  { name: "Judge rubric", file: "evals/judge-rubric.md", done: true },
  { name: "CI gate", file: ".github/workflows/evals.yml", done: false },
  { name: "Tracing", file: "otel/instrument.ts", done: true },
  { name: "Alerts", file: "monitoring/alerts.yaml", done: false },
];

const missing = pack.filter((p) => !p.done);
console.log(`${pack.length - missing.length} of ${pack.length} parts ready`);
for (const p of missing) console.log(`missing: ${p.name} (${p.file})`);
console.log(missing.length ? "not ready to ship" : "ready to ship");
```

Expected output:

```text
5 of 7 parts ready
missing: CI gate (.github/workflows/evals.yml)
missing: Alerts (monitoring/alerts.yaml)
not ready to ship
```

**Try this:** set both missing parts to `done: true`. Predict the last line, then run it.

**About this capstone**

- The incidents, pull requests and costs are simulated, so every run gives the same result and can be graded.
- The exported files are real starting points. Wire `runFeature()` to your app, add your own cases, and fill in your platform's endpoint from its docs.
- The README in the pack has an ungraded checklist for those last steps.

## Your Own Feature

You may not run an online shop. The pack still fits.

For an existing app, start from six questions:

1. **What is the feature's job?** That tells you which real inputs go in the eval set.
2. **What do users complain about?** Those are your first failure categories and code graders.
3. **What is good but hard to check in code?** That goes in the judge rubric.
4. **What labels do you already have?** Use them to calibrate the judge.
5. **What are your targets for time and cost?** Those become dashboard lines and alerts.
6. **What changes often?** That is what the CI gate must run on.

The third exercise below walks through a real answer for each one.

**Start small, then grow**

You do not need all nine parts on day one. Start with 50 real cases, two code graders and one alert with an owner.

Then add a part each time an incident shows you a gap. That is the eval loop at work.

## Where It Is Used

- **Before launching** any AI feature.
- **When you inherit** an AI feature with no evals.
- **In a design review**, as the checklist for "how will we know it works?"
- **After an incident**, to see which part was missing.

**Do's and Don'ts**

- **Do** test your alerts against real or seeded incidents.
- **Do** give every alert an owner and a runbook.
- **Do** keep the pack in your repo, next to the code it tests.
- **Do** add every new failure back into the eval set.
- **Don't** ship a dashboard with no alerts on it.
- **Don't** roll back when nothing of yours changed.
- **Don't** tie your evals to one platform's format.

## Common mistakes

| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| Charts, no alerts | "We have a dashboard, so we will notice." | Put alert rules with owners on the key lines. | Nobody looks at the chart on the day it matters. |
| Rolling back everything | "Something broke, so undo the last deploy." | Roll back only when our change caused it. | You undo a good change and the real cause stays. |
| One rule for everything | "A quality alert covers it." | One rule per metric: quality, cost, latency, errors. | Cost spikes and outages go unnoticed. |
| Untested alerts | "The thresholds look sensible." | Replay incidents and count catches and false alerts. | Alerts fire too late or all the time. |
| Platform lock-in | "Our evals live in the vendor's UI." | Keep evals as files, and send spans with OTel. | Changing platforms means rebuilding everything. |

## Check Your Understanding

## Exercises

## What to Remember

- The pack is the eval loop written as files: **measure, gate, watch**.
- Each part answers one question. A missing part is where incidents hide.
- Test the pack against incidents: count **catches** and **false alerts**.
- **Roll back** what you changed. **Investigate** what you did not. Let the **gate** decide pull requests.
- Keep the pack in your repo, platform-neutral, and feed every new failure back in.

Next: **Cheat Sheet**, every idea from the course on one page.