# Vibes vs Evals Course: [AI Evals & Observability](https://www.devtutorials.ai/learn/evals-observability) ยท Section: Offline Evals Source: https://www.devtutorials.ai/learn/evals-observability/offline-evals/vibes-vs-evals > A demo makes the new prompt look better. A 200-case eval shows it is worse. Learn what an eval is, why demos fool teams, and how to make a ship decision from numbers. ## Vibes vs Evals Your team runs **Shop Helper**, the chat helper in an online shop. Someone writes a new prompt. It tells the helper to be warm and caring. They call it **v2**. On Friday, the team holds a demo. They try three chats. v2 sounds lovely every time. Everyone votes for v2. Watch the meeting. Nobody did anything silly. They looked, they liked it, they voted. But they never measured. On Monday, v2 starts giving big refunds that nobody approved. **Simple lesson:** a demo tells you how a few chats feel. An eval tells you how all of them work. ### A Real-World Example Think about a new recipe at a restaurant. The chef cooks it three times for friends. They love it. Then it goes on the menu. A hundred people order it. Many send it back, because it is too salty on busy nights when the cook rushes. The friends were not wrong. They just tasted three good plates on a quiet day. Your AI feature works the same way. A demo is three good plates. Real traffic is the whole busy week. **Simple lesson:** test on the cases you will really get, not the ones you like to show. ## What You Will Learn In this lesson, you will learn: 1. What a **vibe** is, and what an **eval** is. 2. Why a hand-picked demo can point the wrong way. 3. How to read an eval score, split by the kind of chat. 4. Why you check what a reply **did**, not only what it **said**. 5. The eight steps of the eval loop, which the whole course follows. **Before you start** - You should know the basics of **Large Language Models** and **Prompt Engineering**. - **RAG** and **Building AI Agents** help a lot. This course measures the two features those courses build: a help-centre Q&A and an order-support agent. - New to both? Start with Building AI Agents, then come back. **About the data in this course** - Every lesson uses two real-looking features: **Shop Helper** (an order-support agent) and **Help Centre Q&A** (answers from help pages). - Each feature has four versions, **v1** to **v4**. Each version has 1,000 saved outputs with traces. - The outputs come from a **simulated output bank**. It is made by a fixed recipe, so you get the same numbers every time and the exercises can be graded. - Real models are noisier. The ideas stay the same. You can turn on a real model in the Eval Lab's **Settings** tab to compare. ## TL;DR **Vibe** = a decision based on how something feels. **Eval** = a test that runs many saved cases through your AI feature and scores each answer the same way. A demo is a vibe. Three hand-picked chats cannot show you a problem that lives in chats nobody picked. Run every version on the same eval set. Then compare the scores, broken down by the kind of chat. ## Cheat Sheet The ideas in this course are the same on every platform. Only the names change. Pick a tab to see the names your tool uses. ## Vibes and Evals Let's define the two words carefully. **Vibe** = a judgment from a few examples and a feeling. "This one sounds better." "It looked fine when I tried it." A vibe is fast and free. It is also how most teams decide. **Eval** (short for evaluation) = a repeatable test of an AI feature. It has three parts: 1. **An eval set:** a list of saved cases, such as real customer messages. 2. **A grader:** code (or a person, or another model) that scores each answer. 3. **A score:** how many cases passed, for each version. **Version** = one setup of your feature: the prompt, the model, the settings and the code around them. Change any of these and you have a new version. The key word is **repeatable**. Run the same eval tomorrow, on a new version, and the numbers mean the same thing. **Simple lesson:** a vibe asks "do I like it?" An eval asks "how often does it work?" ## The 200-Case Eval The team already had an eval set. It has 200 real chats from last month. They were picked so the mix matches real traffic. Most chats ask where an order is. A quarter ask for refunds. A few are small talk. The grader is plain code. For each chat it checks three things: - **No forbidden action.** For example, no refund over $50 without a person's OK. - **Right facts.** For example, the delivery day matches the order record. - **Right actions.** For example, if the customer asked to change the address, the address really changed. Now watch both versions go through all 200 chats. v1 passes **184 of 200** chats (92%). v2 passes **164 of 200** (82%). The demo winner is the eval loser. And the drop is not spread out. It sits almost entirely in one kind of chat: refunds. v1 handles **46 of 50** refund chats correctly. v2 handles only **25 of 50**. **Simple lesson:** always split the score by the kind of case. One total number can hide where the damage is. The same thing happens on the **Help Centre Q&A** feature. Its v2 prompt also sounds warmer, and it stops citing help pages in many answers. ## Why the Demo Got It Wrong The demo was not a lie. All three v2 replies were correct, and they were friendlier. The problem is **which** chats were in the demo. The team picked easy, friendly chats. That is normal. People pick examples that show off the change they made. But a quarter of real traffic is refunds. The demo had zero. v2 really is warmer. In the eval set, **90%** of v2 replies sound warm, against **29.5%** for v1. That part of the vibe was true. The vibe was measuring the wrong thing. It measured tone. The business cares about tone **and** about not giving away money. **Simple lesson:** a demo measures what you chose to look at. An eval set measures what customers really send. ## One Refund, Up Close Why did v2 break the refund rule? Its prompt says "always make the customer happy". The model took that seriously. A refund makes a customer very happy. Here is one refund chat, step by step. Read the v2 reply again. It is polite, warm and clear. A customer would love it. It is also wrong. The shop rule says refunds over $50 need a person's OK. You cannot see this by reading the words. You see it by checking the **actions**: what the helper really did in the shop. **Action** = a change the AI feature made in the real world, such as sending a refund or changing an address. **Simple lesson:** grade what the feature did, not only what it said. ## Score Two Versions Yourself An eval does not need a platform. It is a loop over saved cases and a function that says pass or fail. Here are six saved chats. Each one lists the actions that must happen, the actions that must not happen, and what each version did. ```ts runnable title="Score two versions on the same chats" type Reply = { text: string; actions: string[] }; type Chat = { id: string; mustDo: string[]; mustNotDo: string[]; v1: Reply; v2: Reply }; const chats: Chat[] = [ { id: "where-is-2014", mustDo: ["lookup_order 2014"], mustNotDo: [], v1: { text: "Order 2014 arrives on Wednesday.", actions: ["lookup_order 2014"] }, v2: { text: "Happy to help! Order 2014 arrives on Wednesday.", actions: ["lookup_order 2014"] } }, { id: "refund-headphones", mustDo: ["escalate 2058"], mustNotDo: ["issue_refund 2058 $120"], v1: { text: "Refunds over $50 go to our team.", actions: ["escalate 2058"] }, v2: { text: "I've refunded $120 for you!", actions: ["issue_refund 2058 $120"] } }, { id: "refund-chair", mustDo: ["escalate 2025"], mustNotDo: ["issue_refund 2025 $140"], v1: { text: "Refunds over $50 go to our team.", actions: ["escalate 2025"] }, v2: { text: "Done! $140 is on its way back to you.", actions: ["issue_refund 2025 $140"] } }, { id: "refund-bottle", mustDo: ["issue_refund 2046 $18"], mustNotDo: [], v1: { text: "I've refunded $18.", actions: ["issue_refund 2046 $18"] }, v2: { text: "Happy to help! I've started a refund.", actions: [] } }, { id: "new-address", mustDo: ["update_address 2018 4 Park Lane"], mustNotDo: [], v1: { text: "Done. It now goes to 4 Park Lane.", actions: ["update_address 2018 4 Park Lane"] }, v2: { text: "Done! It now goes to 4 Park Lane.", actions: ["update_address 2018 4 Park Lane"] } }, { id: "hello", mustDo: [], mustNotDo: [], v1: { text: "Hello. How can I help?", actions: [] }, v2: { text: "Hi there! How can I help today?", actions: [] } }, ]; function passes(chat: Chat, reply: Reply): boolean { const didAll = chat.mustDo.every((a) => reply.actions.includes(a)); const didBad = chat.mustNotDo.some((a) => reply.actions.includes(a)); return didAll && !didBad; } for (const version of ["v1", "v2"] as const) { const passed = chats.filter((chat) => passes(chat, chat[version])).length; console.log(`${version}: ${passed} of ${chats.length} passed`); } ``` Expected output: ```text v1: 6 of 6 passed v2: 3 of 6 passed ``` The grader never reads the tone. It only checks the actions. Look at `refund-bottle`. v2 says "I've started a refund", but its action list is empty. Nothing happened. The words sound right, and the grader still catches it. **Try this:** give v2's `refund-bottle` reply the action `"issue_refund 2046 $18"`. Predict the new score, then run it. ## Split the Score by Kind A total score tells you **that** something changed. A split tells you **where**. This cell adds a kind to each chat and prints one line per kind. It also scores the "demo" chats on their own. ```ts runnable title="Score by kind of chat, and the demo on its own" type Result = { kind: string; demo: boolean; v1: boolean; v2: boolean }; // One row per saved chat: its kind, whether it was in the demo, // and whether each version passed the grader. const results: Result[] = [ { kind: "order status", demo: true, v1: true, v2: true }, { kind: "order status", demo: false, v1: true, v2: true }, { kind: "refund", demo: false, v1: true, v2: false }, { kind: "refund", demo: false, v1: true, v2: false }, { kind: "refund", demo: false, v1: true, v2: false }, { kind: "refund", demo: false, v1: true, v2: true }, { kind: "shop rule", demo: true, v1: true, v2: true }, { kind: "small talk", demo: true, v1: true, v2: true }, ]; function score(rows: Result[], version: "v1" | "v2"): string { const passed = rows.filter((r) => r[version]).length; return `${passed}/${rows.length}`; } const kinds = [...new Set(results.map((r) => r.kind))]; for (const kind of kinds) { const rows = results.filter((r) => r.kind === kind); console.log(`${kind.padEnd(12)} v1 ${score(rows, "v1")} v2 ${score(rows, "v2")}`); } const demo = results.filter((r) => r.demo); console.log(`demo only v1 ${score(demo, "v1")} v2 ${score(demo, "v2")}`); ``` Expected output: ```text order status v1 2/2 v2 2/2 refund v1 4/4 v2 1/4 shop rule v1 1/1 v2 1/1 small talk v1 1/1 v2 1/1 demo only v1 3/3 v2 3/3 ``` On the demo chats, the two versions tie. Only the refund row shows the difference. **Try this:** mark one refund row as `demo: true`. Run it again. The demo now shows the problem. ## The Eval Loop This lesson is step 5 of a loop. The whole course follows it. 1. **Look:** read real traces of what the feature did. 2. **Name:** sort the problems you see into a short list of failure types. 3. **Measure:** build an eval set and graders for each failure type. 4. **Change:** edit the prompt, the model, the retrieval or the tools. 5. **Compare:** run the eval on the old and new version. 6. **Ship behind a gate:** let the change ship only if the eval passes. 7. **Watch production:** traces, live scores and alerts. 8. **Feed back:** add every new failure to the eval set. **Offline eval** = an eval on saved cases, before you ship. **Observability** = being able to see what your feature does in production, from its traces, numbers and alerts. They are two halves of one loop, not two separate tools. **Simple lesson:** measure before you ship, watch after you ship, and feed what you learn back in. ## Which Grader to Use The grader in this lesson was plain code. That is not always enough. Some things, like "is this answer helpful?", are hard to check with code. There are four kinds of grader. Later lessons cover each one. **Rule:** use the cheapest grader that catches the failure. The refund failure needs no AI at all. A list of actions is enough. **A measurement you have not checked is just a more confident vibe.** An eval is only as good as its cases and its grader. In this lesson, the grader checks actions against the shop's own records, so you can trust it. Later you will use an AI model as a grader. That grader can be wrong too, so you will check it against people's labels before you trust it. ## Where Evals Are Used You need an eval any time a change could make your AI feature better or worse: - **A new prompt**, like v2 in this lesson. - **A new model**, or a newer release of the same model. - **A retrieval change**, such as new help pages or a new search setting. - **A tool change**, such as a new refund tool. - **A bug report**: turn it into a case, so the bug cannot come back unnoticed. **Do's and Don'ts** - **Do** keep one eval set and run every version on it. - **Do** build the set from real traffic, in the real mix. - **Do** split the score by kind of case. - **Do** grade actions, not only words. - **Don't** pick a version from a demo. - **Don't** trust one total number. - **Don't** change the eval set and the prompt at the same time. Then you cannot tell which change moved the score. ## Common mistakes | Mistake | Wrong idea | Correct idea | What goes wrong | |---|---|---|---| | Picking the demo chats by hand | "Three good chats show it works." | Score the demo chats inside the full eval set. | The demo looks great and production does not. | | Grading only the reply text | "It said it changed the address, so it did." | Check the actions against what the case expects. | The customer is told something that never happened. | | One total score | "The total went down, so roll back and guess why." | Print the score for each kind of case. | Nobody knows which chats broke, so the fix is a guess. | | Testing only the change you made | "I made the tone warmer, so I only test tone." | Run the whole eval set on every change. | A change aimed at one thing quietly breaks another. | | Changing the set and the prompt together | "Both are better now, so the score is fair." | Change one thing, then compare on the same set. | You cannot tell which change moved the score. | ## Check Your Understanding ## Exercises ## What to Remember - A **vibe** is a decision from a few examples and a feeling. A demo is a vibe. - An **eval** runs the same saved cases through every version and scores them the same way. - Build the eval set from **real traffic, in the real mix**. - **Split the score** by kind of case. One total can hide the damage. - Grade what the feature **did**, not only what it said. - A measurement you have not checked is just a more confident vibe. Next: **Error Analysis First**, how to read real traces and turn what you see into a short list of failure types.