# Capstone: Ship the Claims Pipeline

Course: [Multimodal AI](https://www.devtutorials.ai/learn/multimodal-ai) · Section: Listening, Speaking and Generating
Source: https://www.devtutorials.ai/learn/multimodal-ai/listening-and-generating/capstone

> Run every lesson's settings on a held-out set, pass five axes (accuracy, verification, cost, latency and privacy), and export the full pipeline.

## What the Capstone Is

**Capstone** = the final project. You run the whole claims pipeline you built, with every lesson's settings, on items it has never seen.

Each lesson tested one stage on its own items. That is not enough. Settings can be tuned until they pass a small test, and then fail on the next receipt. This is why the capstone uses a **held-out set**: 4 receipts, 3 invoices, 6 extractions and 4 voice notes that no lesson used.

Press **Play**, or use **Next** to go one step at a time.

- **Frame 1:** every stage, with the lesson that set it.
- **Frame 2:** the scorecard: five axes, each with its target.

**Simple lesson:** a pipeline is ready when it passes on items it has never seen.

### A Real-World Example

Think about a driving test.

- You practise on roads you know, again and again.
- The examiner takes you on roads you have never driven.
- They score you on several things: control, safety, signals, speed.
- Failing one is enough to fail the test, and they tell you which one.

The capstone works the same way:

- **Roads you know** = each lesson's own items.
- **New roads** = the held-out set.
- **Several scores** = accuracy, verification, cost, latency and privacy.
- **Which one failed** = the scorecard points you back to the lesson to fix.

**Simple lesson:** test on new items, on every axis, and fix the one that fails.

## What You Will Learn

By the end of this lesson, you will be able to:

- set targets before you test
- run a held-out evaluation
- read a five-axis scorecard
- trace a failing axis back to its lesson
- export the full pipeline

## First: Five Questions

Before you run anything, answer five questions about your own app. The answers become `PLAN.md` in your export.

Step through both frames.

- **Frame 1:** the five questions and the reference answers.
- **Frame 2:** with no answers, even perfect settings do not pass.

| Question | Why it matters |
|---|---|
| Which media will your app take in? | Decides which stages you need |
| Which fields are critical? | Decides what must be checked |
| What accuracy bar must automatic values meet? | Decides how much goes to review |
| How fast must a claim be processed? | Decides cloud or device, streaming or batch |
| What may leave the user's device? | Decides where each model runs |

**Targets chosen after you see the scores are not targets.** If you pick the bar after the run, you will pick whatever passed. Decide first.

## Five Axes

| Axis | Target | Lesson |
|---|---|---|
| Accuracy | at least 95% on receipts, invoices and speech | 2, 3, 7 |
| Verification coverage | 0 silent totals, at least 90% of wrong fields caught, at most 10% false alarms | 5 |
| Cost | at most 8¢ per claim | 2, 13 |
| Latency | at most 15 s per claim | 3, 7, 13 |
| Privacy | 0 pieces of personal data uploaded | 13 |

These are the defaults. **Your answers change them:**

| Your answer | What changes |
|---|---|
| Accuracy bar 98% or 99% | The accuracy target rises to match |
| "100%: every value is reviewed" | Every wrong field must be caught; false alarms are allowed |
| "Under 1 minute" or slower | The latency target relaxes |
| Media: voice notes only, or receipts and invoices only | Only those parts count for accuracy |
| "Anything, under a data agreement" | Uploads are allowed, but still logged |

The reference settings pass every axis: 100% accuracy, 4 of 4 wrong fields caught with 7% false alarms, about 1.5¢ and 8.1 s per claim, and no personal data uploaded.

Step through both frames.

- **A tiny speech model with no fixes:** Accuracy fails.
- **The receipt photo sent to the cloud:** Privacy fails.

**One shortcut fails one axis.** Do not lower a target to make it pass. Go back to the lesson that owns the setting, fix it there, and run the capstone again.

**Easy way to remember:** questions first, held-out set, five green rows, then export.

### Lab Exercise: Run the Capstone

Your task: answer the five questions in the **Capstone** tab, then press **Run the capstone**.

Click the button and open the **Capstone** tab.

- **The questions:** one answer each. The targets they set appear under the questions.
- **Follow one held-out claim:** see the claim at each of the six stages, with your settings.
- **The scorecard:** each axis, its value and its target.
- **To pass:** all five axes green, with 0 silent errors on totals.

## Export the Full Pipeline

Click the button and open the **Export** tab.

- **The table:** every part of the pipeline, and whether it passes.
- **The zip:** `multimodal-pipeline.zip` when everything is ready.

The export adds these files to your v1 pipeline from Lesson 6:

| File | What it holds |
|---|---|
| `src/speech.ts` | Your speech settings and `normalizeNumbers()` |
| `src/turnTaking.ts` | Your endpointing and barge-in settings |
| `src/turnTaking.check.ts` | Your `shouldEndTurn()` |
| `src/imageStudio.ts` | Your image studio settings and `judge()` |
| `src/sampleFrames.ts` | Your `sampleFrames()` and video settings |
| `src/videoGeneration.ts` | Your video generation settings |
| `src/routeQuery.ts` | Your search routing and citation settings |
| `src/redact.ts` | Your `redact()` rules |
| `PLAN.md` | Your five answers and the targets |

Run the eval harness in the zip before you connect a real model. Then swap the seed bank for your own provider, and run it again.

## TL;DR

1. Test on a **held-out set**, not the lessons' own items.
2. Answer the **five questions** first; they set the targets.
3. Five axes: **accuracy, verification, cost, latency, privacy**.
4. All five must pass, with **0 silent totals**.
5. A failing axis points you to **its lesson**.
6. **Export**, then run the harness with your real model.

## Common mistakes

| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| Testing on the lessons' items | "It passed already." | Use the held-out set. | It fails on the first new receipt. |
| Setting targets after | "Let's see what we get." | Answer the questions first. | Any result looks good enough. |
| Lowering a target | "8¢ is close to 10¢." | Fix the setting in its lesson. | The pipeline ships over budget. |
| Watching one axis | "Accuracy is 100%." | All five must pass. | Personal data leaks while accuracy is perfect. |
| Skipping the harness | "The lab passed." | Run it with your real model. | The real model behaves differently. |

## Check Your Understanding

## Exercises

## What to Remember

- Decide the targets first.
- Test on items you have never seen.
- Every axis must pass.
- Fix a failing axis in its own lesson.
- Export, then test again with your real model.

## Further Reading

- [OpenAI Evals, by OpenAI](https://github.com/openai/evals): a framework for running held-out evaluations.
- [People + AI Guidebook, by Google PAIR](https://pair.withgoogle.com/guidebook/): designing for errors and human control.

Next: **Multimodal AI Cheat Sheet**, with every rule and target from the course on one page.