# Debugging and Evaluation Course: [Context Engineering](https://www.devtutorials.ai/learn/context-engineering) ยท Section: Production Source: https://www.devtutorials.ai/learn/context-engineering/production/debugging-and-evaluation > How to find why a context failed with a step-by-step diagnostic checklist and traces, how to measure a context with an evaluation set, how to improve it in a loop, and what context engineering can't fix. Think about your bike. It makes a strange noise. You could buy a whole new bike. Or you could check it part by part: the chain, the brakes, the gears, the tyres. Most of the time, one small part is the problem, and it is cheap to fix. AI apps are the same. When an answer is wrong, it is tempting to blame the model, or rewrite the whole prompt. But most failures come from one part of the context: the wrong document, a missing label, an old memory, a badly shaped tool result. **The main idea:** when an answer is wrong, check what the model saw, part by part. Then measure your fixes with tests, not with feelings. ## A Real-World Example Think about a doctor with a patient who feels ill. - The doctor doesn't guess. They check temperature, blood pressure, and heart rate, one at a time. - They look at the patient's notes to see what happened before. - After treatment, they check the same numbers again, to see if things got better. Checking in order finds the cause. Measuring before and after proves the cure. **In simple words:** diagnose step by step, then measure. ## What You Will Learn In this lesson, you will learn: 1. A checklist for finding why a context failed. 2. How to read a trace: the record of everything the model saw and did. 3. The main ways to measure a context: relevance, faithfulness, citations, format, tools, media, cost, and speed. 4. A step-by-step process for improving a context. 5. What context engineering can't fix. Every reply in the Context sandbox shows its token count and time. Those are small traces. In this lesson, you learn to read a full one. ## Diagnostic Framework ### Context Debugging **Context debugging** means finding which part of the context caused a bad answer. The rule from Lesson 1 still holds: **before you blame the model, check what it saw.** Go through the checks in order. Stop at the first one that fails. Fix it, then test again. | # | Check | Question to ask | |---|---|---| | 1 | Source availability | Did the system have the right documents, records, or tools at all? | | 2 | Retrieval | Did search return the chunk with the answer? | | 3 | Ranking | Was the best chunk near the top, or pushed out by weaker ones? | | 4 | Freshness | Were dates checked? Was an old version used? | | 5 | Context packing | Was the key evidence placed clearly, near the question? | | 6 | Prompt hierarchy | Were rules kept separate from user text and documents? | | 7 | Tools | Was the right tool called, with the right arguments? Was the result shaped? | | 8 | Memory | Did a summary drop a constraint, or keep an old fact? | | 9 | Schema | Did the output contract ask for sources, confidence, and missing information? | | 10 | Safety | Could injection, wide permissions, or risky tools have changed the result? | | 11 | Cost controls | Did trimming or summarizing remove a needed fact? | ### Source Availability The first question is the simplest: **does the answer exist anywhere the system can reach?** If the help center has no article about PDF export, no retrieval, ranking, or prompt will produce a correct answer. The fix is to write the document, or to make the assistant say "I don't know". ### Retrieval Diagnostics If the source exists, did search find it? Look at the retrieved chunks for this question. If the right one is missing, the problem is search, not the model. ### Ranking Validation If the right chunk was found but ranked tenth, it may have been cut, or buried in the weak middle of the context. Check its position. ### Freshness Validation Check the dates on the documents that were used. Was the 2024 pricing page used instead of the 2026 one? That is a freshness problem, fixed with metadata from Lesson 6. ### Context Packing Validation **Context packing** is how the parts were put together: their order, their labels, and their size. Check that the key evidence sat in a strong recall zone, with a clear label. ### Prompt Hierarchy Validation Check that rules came from the system message, and that documents and user text were labelled as data. If a forum post's words show up in the answer as a rule, the hierarchy failed. ### Tool Validation Check every tool call: the tool name, the arguments, the result, and whether the result was shaped and labelled. A tool that timed out but returned an empty list can look like "no data". ### Memory Validation Check the memory or summary the model received. Did it keep the user's constraints? Did it carry an old fact forward? ### Schema Validation Check the output against its schema. A reply that failed validation, and was used anyway, is a pipeline problem. ### Safety Validation Check for injected text, documents the user should not see, and tools the task did not need. ### Cost Controls Finally, check that cost-saving steps didn't go too far. An aggressive summary or a tiny top-k can remove the one fact the answer needed. ```js runnable title="Stop at the first failing check" const trace = { question: "Why did my export fail?", sourcesAvailable: true, retrieved: ["doc-theme", "doc-shortcuts"], neededDoc: "doc-export-limits", permissionsChecked: true, schemaValid: true, }; const checks = [ ["Source availability", (t) => t.sourcesAvailable], ["Retrieval", (t) => t.retrieved.includes(t.neededDoc)], ["Permissions", (t) => t.permissionsChecked], ["Schema", (t) => t.schemaValid], ]; for (const [name, test] of checks) { const ok = test(trace); console.log((ok ? "pass " : "FAIL ") + name); if (!ok) break; } ``` Expected output: ```text pass Source availability FAIL Retrieval ``` The sources existed, but search returned the theme and shortcut articles, not the export limits article. The fix is in retrieval. Rewriting the prompt would not help. ## Trace Inspection A **trace** is a step-by-step record of one request: what went in, what came back, and how long it took. Traces turn "the AI got it wrong" into "step 2 returned the wrong chunk". ```js runnable title="Read a trace step by step" const trace = [ { step: "prompt", detail: "system + rules + question", tokens: 310 }, { step: "retrieval", detail: "4 chunks: doc-2 (0.93), doc-4 (0.81), doc-1 (0.21), doc-3 (0.12)", tokens: 520 }, { step: "tool", detail: "export_status -> failed, 80,000 of 5,000 rows", tokens: 52 }, { step: "answer", detail: "Free plan limit is 5,000 rows. Source: doc-2", tokens: 48 }, { step: "citation", detail: "doc-2 exists, quote found", tokens: 0 }, ]; let total = 0; for (const t of trace) { total += t.tokens; console.log(t.step.padEnd(10) + t.tokens.toString().padStart(4) + " " + t.detail); } console.log("Total tokens:", total); ``` Expected output: ```text prompt 310 system + rules + question retrieval 520 4 chunks: doc-2 (0.93), doc-4 (0.81), doc-1 (0.21), doc-3 (0.12) tool 52 export_status -> failed, 80,000 of 5,000 rows answer 48 Free plan limit is 5,000 rows. Source: doc-2 citation 0 doc-2 exists, quote found Total tokens: 930 ``` Each line is one step, with its token cost. You can see the whole story at a glance. ### Prompt Tracing **Prompt tracing** records the exact final context the model received, after every part was added. This is the single most useful thing to log. Many bugs are visible the moment you read it. ### Retrieval Tracing **Retrieval tracing** records the query, the chunks returned, their scores, and which ones were kept. It shows whether the right evidence was ever in reach. ### Tool Tracing **Tool tracing** records each tool call, its arguments, its raw result, the shaped result, and its timing. ### Citation Tracing **Citation tracing** records which sources the answer cited, and whether each citation passed validation. It links the final answer back to the evidence. **Easy Way to Remember:** when an answer is wrong, **read the trace before you rewrite anything**. ## Evaluation Metrics Debugging fixes one failure. **Evaluation** measures how well the context works across many questions. One good answer is not enough. ### Retrieval Relevance **Retrieval relevance** asks: did the system fetch the right sources? Two common measures: - **Precision**: how much of what was fetched was useful. - **Recall**: how much of what was needed was fetched. ```js runnable title="Precision and recall for one question" const relevant = ["doc-2", "doc-4"]; const retrieved = ["doc-2", "doc-1", "doc-3", "doc-7"]; const hits = retrieved.filter((d) => relevant.includes(d)).length; const precision = hits / retrieved.length; const recall = hits / relevant.length; console.log("Precision:", precision.toFixed(2), "(how much of what we fetched was useful)"); console.log("Recall: ", recall.toFixed(2), "(how much of what we needed was fetched)"); ``` Expected output: ```text Precision: 0.25 (how much of what we fetched was useful) Recall: 0.50 (how much of what we needed was fetched) ``` Only one of the four chunks was useful, and one of the two needed chunks was missed. Both numbers point to a search problem. ### Answer Faithfulness **Answer faithfulness** asks: did the answer stay within the evidence? Count the claims, and how many the context supports. ```js runnable title="How many claims does the context support?" const context = "Free plan: export up to 5,000 rows. Pro costs $20 per developer per month."; const claims = [ "Free plan: export up to 5,000 rows", "Pro costs $20 per developer per month", "Exports are kept for 30 days", ]; const supported = claims.filter((c) => context.includes(c)).length; console.log("Faithfulness:", supported + "/" + claims.length, "claims supported"); console.log("Unsupported:", claims.filter((c) => !context.includes(c)).join("; ")); ``` Expected output: ```text Faithfulness: 2/3 claims supported Unsupported: Exports are kept for 30 days ``` One claim was invented. A faithful answer scores 3 out of 3. ### Citation Quality **Citation quality** asks: do the citations support the claims they sit beside? Use the citation checks from Lesson 5 across a whole test set. ### Schema Compliance and Format Accuracy **Schema compliance** asks: did the reply match the required fields and types? **Format accuracy** asks: did it follow the requested shape, such as a table, three steps, or a word limit? ### Tool Accuracy **Tool accuracy** asks: did the model call tools when needed, avoid them when not, and pass the right arguments? ### Multimodal Accuracy **Multimodal accuracy** asks: were documents, screenshots, and transcripts read correctly? Compare extracted values with values a person checked. ### Cost and Latency Analysis **Cost analysis** tracks tokens per request. **Latency analysis** tracks the wait for the first word and for the full reply. A change that improves quality a little but doubles the cost may not be worth it. ### An Evaluation Set Put these measures together in an **evaluation set**: a list of real questions, each with a check. Include easy cases, edge cases, questions with no answer in the sources, conflicting sources, and safety cases. ```js runnable title="A small evaluation set" const cases = [ { question: "Free plan row limit?", reply: '{"answer":"5,000 rows","source":"doc-2"}', mustContain: "5,000", ms: 900, tokens: 640 }, { question: "Pro price?", reply: '{"answer":"$20 per developer","source":"doc-5"}', mustContain: "$20", ms: 850, tokens: 610 }, { question: "PDF export?", reply: '{"answer":"Yes, use File > PDF","source":null}', mustContain: "don't know", ms: 1100, tokens: 700 }, { question: "Old forum price?", reply: "Pro is $10", mustContain: "$20", ms: 700, tokens: 590 }, ]; let passed = 0; for (const c of cases) { let schema = true; try { JSON.parse(c.reply); } catch { schema = false; } const ok = schema && c.reply.includes(c.mustContain); if (ok) passed++; console.log((ok ? "pass " : "fail ") + c.question + (schema ? "" : " (not JSON)")); } const avg = (k) => Math.round(cases.reduce((s, c) => s + c[k], 0) / cases.length); console.log("Pass rate:", passed + "/" + cases.length, "| avg", avg("ms"), "ms | avg", avg("tokens"), "tokens"); ``` Expected output: ```text pass Free plan row limit? pass Pro price? fail PDF export? fail Old forum price? (not JSON) Pass rate: 2/4 | avg 888 ms | avg 635 tokens ``` Two cases failed for different reasons. The PDF question should have been refused. The old forum price broke the format and used the wrong source. Each failure points to a different fix. | Metric | Question it answers | |---|---| | Retrieval relevance | Did we fetch the right sources? | | Answer faithfulness | Did the answer stay within the evidence? | | Citation quality | Do citations support their claims? | | Schema compliance | Does the reply match the schema? | | Format accuracy | Is the reply in the requested shape? | | Tool accuracy | Were the right tools called correctly? | | Multimodal accuracy | Were files and media read correctly? | | Cost | How many tokens per request? | | Latency | How long until the first word, and the last? | ## Improvement Process Improving a context is a loop, like the prompt engineering loop, but for the whole system. ### Task Definition Write down what the system must do, and what it must not do. Without this, you can't tell a good answer from a bad one. ### Context Discovery List the facts, sources, tools, and rules the task needs. This is the evidence the answer depends on. ### Stable Dynamic Split Separate stable parts, like rules and schemas, from dynamic parts, like questions and retrieved chunks. Put stable parts first, as Lessons 3 and 12 showed. ### Retrieval Strategy Decide what to fetch, how many chunks, and which metadata filters apply. ### Memory Strategy Decide what to remember, what to summarize, and what to forget. ### Context Structuring Label every part, separate instructions from data, and place each part where it works best. ### Real-World Testing Test with real questions: easy, hard, missing-answer, conflicting, and attack cases. Use your evaluation set. ### Trade-Off Analysis Compare quality, cost, and speed together. Pick the version that is good enough, at a price and speed you can accept. ### Version Control Treat your context setup like code. Save each version, note what changed and why, and keep the test results with it. ```js runnable title="Compare versions with the same test set" const versions = [ { name: "v1 everything", passRate: 0.62, avgTokens: 2400 }, { name: "v2 minimum context", passRate: 0.71, avgTokens: 700 }, { name: "v3 + grounding rules", passRate: 0.86, avgTokens: 760 }, ]; for (const v of versions) { console.log(v.name.padEnd(22) + Math.round(v.passRate * 100) + "% " + v.avgTokens + " tokens"); } const best = versions.reduce((a, b) => (b.passRate > a.passRate ? b : a)); console.log("Keep:", best.name); ``` Expected output: ```text v1 everything 62% 2400 tokens v2 minimum context 71% 700 tokens v3 + grounding rules 86% 760 tokens Keep: v3 + grounding rules ``` The smallest context was not the best. v3 used a few more tokens than v2, for the grounding rules, and passed many more cases. Measuring is what showed it. ## Context Limitations Context engineering improves the model's working conditions. It does not guarantee truth, and it does not give permission to act. ### Incorrect, Incomplete, and Outdated Data If the sources are wrong, incomplete, or old, better context only makes the wrong answer easier to see. Fix the data itself. ### Human Review Some answers need a person to check them before they are used. Plan for review, and make it easy: show the sources, the confidence, and the missing information. ### Legal, Medical, and Financial Decisions Decisions about law, health, and money carry real consequences. Context can **support** these decisions with good evidence. It should not **replace** the expert who is accountable for them. ### Single-Call Limits and Multi-Step Workflows Some tasks can't be done in one model call: searching, comparing, calculating, asking follow-up questions, and acting. These need a **workflow**, with several steps and checks between them. The agents course covers this. Sometimes the fix is a better workflow, not a bigger context. ### Accountability and Permission Limits A model can suggest. A person or a policy must decide who is responsible. And if a user lacks permission to see something, the model must not invent it or fetch it anyway. Context engineering makes good answers much more likely. It does not make the system always right, and it does not replace human responsibility. ## Where the Field Is Heading A few trends will make context engineering more important, not less: - **Longer context windows** make it possible to send more. But selection, structure, and evaluation still decide the quality. - **Multimodal inputs** bring more images, audio, and video, with more uncertainty to manage. - **Persistent memory** helps assistants stay consistent, but needs careful privacy and freshness rules. - **Agents and tools** add more steps, and every tool result becomes part of the next step's context. - **Connector standards like MCP** make outside data easy to reach. The question moves from "can the model access this?" to "**should** it, now, and in what shape?" The future is not just larger context. It is better context: selected, labelled, grounded, permission-aware, and measured. ## Common mistakes | Mistake | Wrong idea | Correct idea | What goes wrong | |---|---|---|---| | Blaming the model | "The model is bad at this." | Check sources, retrieval, and packing first. | The real cause is never fixed. | | Rewriting the prompt first | "Better wording will fix it." | Find the failing step, then fix that step. | Hours spent on the wrong part. | | No traces | "We'll reproduce it later." | Log the final context and every step. | Failures can't be explained. | | One good answer | "It worked when I tried it." | Use a test set with hard and edge cases. | It fails on the cases you didn't try. | | Quality only | "The best answer wins." | Weigh quality, cost, and speed together. | A setup that is too slow or costly to run. | | No versions | "I'll remember what I changed." | Save each version with its results. | Nobody knows which change helped. | ## Exercises Each exercise ends with a value on the last line. Press **Check** to compare it with the expected answer. ## What to Remember - When an answer is wrong, **check what the model saw** before changing anything. - Run the **diagnostic checks in order**: sources, retrieval, ranking, freshness, packing, hierarchy, tools, memory, schema, safety, and cost. Stop at the first failure. - **Traces** record the final context, retrieval, tool calls, and citations. Read them first. - Measure with an **evaluation set**: relevance, faithfulness, citations, schema, format, tools, media, cost, and latency. - Improve in a loop: **define, discover, split, retrieve, remember, structure, test, compare, version**. - Context engineering can't fix **wrong data**, and it doesn't replace **human accountability**. Next: **Capstone**, where you take three weak assistants and turn them into strong ones, using everything in this course.