Debugging and Evaluation

By the Syncfusion DevTutorials team · Last updated October 20, 2026

How to find why a context failed with a step-by-step diagnostic checklist and traces, how to measure a context with an evaluation set, how to improve it in a loop, and what context engineering can't fix.

Think about your bike. It makes a strange noise. You could buy a whole new bike. Or you could check it part by part: the chain, the brakes, the gears, the tyres. Most of the time, one small part is the problem, and it is cheap to fix.

AI apps are the same. When an answer is wrong, it is tempting to blame the model, or rewrite the whole prompt. But most failures come from one part of the context: the wrong document, a missing label, an old memory, a badly shaped tool result.

The main idea: when an answer is wrong, check what the model saw, part by part. Then measure your fixes with tests, not with feelings.

A Real-World Example

Think about a doctor with a patient who feels ill.

• The doctor doesn't guess. They check temperature, blood pressure, and heart rate, one at a time.
• They look at the patient's notes to see what happened before.
• After treatment, they check the same numbers again, to see if things got better.

Checking in order finds the cause. Measuring before and after proves the cure.

In simple words: diagnose step by step, then measure.

What You Will Learn

In this lesson, you will learn:

1. A checklist for finding why a context failed.
2. How to read a trace: the record of everything the model saw and did.
3. The main ways to measure a context: relevance, faithfulness, citations, format, tools, media, cost, and speed.
4. A step-by-step process for improving a context.
5. What context engineering can't fix.

Diagnostic Framework

Context Debugging

Context debugging means finding which part of the context caused a bad answer. The rule from Lesson 1 still holds: before you blame the model, check what it saw.

Go through the checks in order. Stop at the first one that fails. Fix it, then test again.

Source Availability

The first question is the simplest: does the answer exist anywhere the system can reach? If the help center has no article about PDF export, no retrieval, ranking, or prompt will produce a correct answer. The fix is to write the document, or to make the assistant say "I don't know".

Retrieval Diagnostics

If the source exists, did search find it? Look at the retrieved chunks for this question. If the right one is missing, the problem is search, not the model.

Ranking Validation

If the right chunk was found but ranked tenth, it may have been cut, or buried in the weak middle of the context. Check its position.

Freshness Validation

Check the dates on the documents that were used. Was the 2024 pricing page used instead of the 2026 one? That is a freshness problem, fixed with metadata from Lesson 6.

Context Packing Validation

Context packing is how the parts were put together: their order, their labels, and their size. Check that the key evidence sat in a strong recall zone, with a clear label.

Prompt Hierarchy Validation

Check that rules came from the system message, and that documents and user text were labelled as data. If a forum post's words show up in the answer as a rule, the hierarchy failed.

Tool Validation

Check every tool call: the tool name, the arguments, the result, and whether the result was shaped and labelled. A tool that timed out but returned an empty list can look like "no data".

Memory Validation

Check the memory or summary the model received. Did it keep the user's constraints? Did it carry an old fact forward?

Schema Validation

Check the output against its schema. A reply that failed validation, and was used anyway, is a pipeline problem.

Safety Validation

Check for injected text, documents the user should not see, and tools the task did not need.

Cost Controls

Finally, check that cost-saving steps didn't go too far. An aggressive summary or a tiny top-k can remove the one fact the answer needed.

Stop at the first failing check

const trace = {
  question: "Why did my export fail?",
  sourcesAvailable: true,
  retrieved: ["doc-theme", "doc-shortcuts"],
  neededDoc: "doc-export-limits",
  permissionsChecked: true,
  schemaValid: true,
};

const checks = [
  ["Source availability", (t) => t.sourcesAvailable],
  ["Retrieval", (t) => t.retrieved.includes(t.neededDoc)],
  ["Permissions", (t) => t.permissionsChecked],
  ["Schema", (t) => t.schemaValid],
];

for (const [name, test] of checks) {
  const ok = test(trace);
  console.log((ok ? "pass  " : "FAIL  ") + name);
  if (!ok) break;
}

Expected output:

pass  Source availability
FAIL  Retrieval

The sources existed, but search returned the theme and shortcut articles, not the export limits article. The fix is in retrieval. Rewriting the prompt would not help.

Trace Inspection

A trace is a step-by-step record of one request: what went in, what came back, and how long it took. Traces turn "the AI got it wrong" into "step 2 returned the wrong chunk".

Read a trace step by step

const trace = [
  { step: "prompt", detail: "system + rules + question", tokens: 310 },
  { step: "retrieval", detail: "4 chunks: doc-2 (0.93), doc-4 (0.81), doc-1 (0.21), doc-3 (0.12)", tokens: 520 },
  { step: "tool", detail: "export_status -> failed, 80,000 of 5,000 rows", tokens: 52 },
  { step: "answer", detail: "Free plan limit is 5,000 rows. Source: doc-2", tokens: 48 },
  { step: "citation", detail: "doc-2 exists, quote found", tokens: 0 },
];

let total = 0;
for (const t of trace) {
  total += t.tokens;
  console.log(t.step.padEnd(10) + t.tokens.toString().padStart(4) + "  " + t.detail);
}
console.log("Total tokens:", total);

Expected output:

prompt     310  system + rules + question
retrieval  520  4 chunks: doc-2 (0.93), doc-4 (0.81), doc-1 (0.21), doc-3 (0.12)
tool        52  export_status -> failed, 80,000 of 5,000 rows
answer      48  Free plan limit is 5,000 rows. Source: doc-2
citation     0  doc-2 exists, quote found
Total tokens: 930

Each line is one step, with its token cost. You can see the whole story at a glance.

Prompt Tracing

Prompt tracing records the exact final context the model received, after every part was added. This is the single most useful thing to log. Many bugs are visible the moment you read it.

Retrieval Tracing

Retrieval tracing records the query, the chunks returned, their scores, and which ones were kept. It shows whether the right evidence was ever in reach.

Tool Tracing

Tool tracing records each tool call, its arguments, its raw result, the shaped result, and its timing.

Citation Tracing

Citation tracing records which sources the answer cited, and whether each citation passed validation. It links the final answer back to the evidence.

Evaluation Metrics

Debugging fixes one failure. Evaluation measures how well the context works across many questions. One good answer is not enough.

Retrieval Relevance

Retrieval relevance asks: did the system fetch the right sources? Two common measures:

• Precision: how much of what was fetched was useful.
• Recall: how much of what was needed was fetched.

Precision and recall for one question

const relevant = ["doc-2", "doc-4"];
const retrieved = ["doc-2", "doc-1", "doc-3", "doc-7"];

const hits = retrieved.filter((d) => relevant.includes(d)).length;
const precision = hits / retrieved.length;
const recall = hits / relevant.length;

console.log("Precision:", precision.toFixed(2), "(how much of what we fetched was useful)");
console.log("Recall:   ", recall.toFixed(2), "(how much of what we needed was fetched)");

Expected output:

Precision: 0.25 (how much of what we fetched was useful)
Recall:    0.50 (how much of what we needed was fetched)

Only one of the four chunks was useful, and one of the two needed chunks was missed. Both numbers point to a search problem.

Answer Faithfulness

Answer faithfulness asks: did the answer stay within the evidence? Count the claims, and how many the context supports.

How many claims does the context support?

const context = "Free plan: export up to 5,000 rows. Pro costs $20 per developer per month.";

const claims = [
  "Free plan: export up to 5,000 rows",
  "Pro costs $20 per developer per month",
  "Exports are kept for 30 days",
];

const supported = claims.filter((c) => context.includes(c)).length;
console.log("Faithfulness:", supported + "/" + claims.length, "claims supported");
console.log("Unsupported:", claims.filter((c) => !context.includes(c)).join("; "));

Expected output:

Faithfulness: 2/3 claims supported
Unsupported: Exports are kept for 30 days

One claim was invented. A faithful answer scores 3 out of 3.

Citation Quality

Citation quality asks: do the citations support the claims they sit beside? Use the citation checks from Lesson 5 across a whole test set.

Schema Compliance and Format Accuracy

Schema compliance asks: did the reply match the required fields and types? Format accuracy asks: did it follow the requested shape, such as a table, three steps, or a word limit?

Tool Accuracy

Tool accuracy asks: did the model call tools when needed, avoid them when not, and pass the right arguments?

Multimodal Accuracy

Multimodal accuracy asks: were documents, screenshots, and transcripts read correctly? Compare extracted values with values a person checked.

Cost and Latency Analysis

Cost analysis tracks tokens per request. Latency analysis tracks the wait for the first word and for the full reply. A change that improves quality a little but doubles the cost may not be worth it.

An Evaluation Set

Put these measures together in an evaluation set: a list of real questions, each with a check. Include easy cases, edge cases, questions with no answer in the sources, conflicting sources, and safety cases.

A small evaluation set

const cases = [
  { question: "Free plan row limit?", reply: '{"answer":"5,000 rows","source":"doc-2"}', mustContain: "5,000", ms: 900, tokens: 640 },
  { question: "Pro price?", reply: '{"answer":"$20 per developer","source":"doc-5"}', mustContain: "$20", ms: 850, tokens: 610 },
  { question: "PDF export?", reply: '{"answer":"Yes, use File > PDF","source":null}', mustContain: "don't know", ms: 1100, tokens: 700 },
  { question: "Old forum price?", reply: "Pro is $10", mustContain: "$20", ms: 700, tokens: 590 },
];

let passed = 0;
for (const c of cases) {
  let schema = true;
  try { JSON.parse(c.reply); } catch { schema = false; }
  const ok = schema && c.reply.includes(c.mustContain);
  if (ok) passed++;
  console.log((ok ? "pass " : "fail ") + c.question + (schema ? "" : " (not JSON)"));
}
const avg = (k) => Math.round(cases.reduce((s, c) => s + c[k], 0) / cases.length);
console.log("Pass rate:", passed + "/" + cases.length, "| avg", avg("ms"), "ms | avg", avg("tokens"), "tokens");

Expected output:

pass Free plan row limit?
pass Pro price?
fail PDF export?
fail Old forum price? (not JSON)
Pass rate: 2/4 | avg 888 ms | avg 635 tokens

Two cases failed for different reasons. The PDF question should have been refused. The old forum price broke the format and used the wrong source. Each failure points to a different fix.

Improvement Process

Improving a context is a loop, like the prompt engineering loop, but for the whole system.

Task Definition

Write down what the system must do, and what it must not do. Without this, you can't tell a good answer from a bad one.

Context Discovery

List the facts, sources, tools, and rules the task needs. This is the evidence the answer depends on.

Stable Dynamic Split

Separate stable parts, like rules and schemas, from dynamic parts, like questions and retrieved chunks. Put stable parts first, as Lessons 3 and 12 showed.

Retrieval Strategy

Decide what to fetch, how many chunks, and which metadata filters apply.

Memory Strategy

Decide what to remember, what to summarize, and what to forget.

Context Structuring

Label every part, separate instructions from data, and place each part where it works best.

Real-World Testing

Test with real questions: easy, hard, missing-answer, conflicting, and attack cases. Use your evaluation set.

Trade-Off Analysis

Compare quality, cost, and speed together. Pick the version that is good enough, at a price and speed you can accept.

Version Control

Treat your context setup like code. Save each version, note what changed and why, and keep the test results with it.

Compare versions with the same test set

const versions = [
  { name: "v1 everything", passRate: 0.62, avgTokens: 2400 },
  { name: "v2 minimum context", passRate: 0.71, avgTokens: 700 },
  { name: "v3 + grounding rules", passRate: 0.86, avgTokens: 760 },
];

for (const v of versions) {
  console.log(v.name.padEnd(22) + Math.round(v.passRate * 100) + "%  " + v.avgTokens + " tokens");
}
const best = versions.reduce((a, b) => (b.passRate > a.passRate ? b : a));
console.log("Keep:", best.name);

Expected output:

v1 everything         62%  2400 tokens
v2 minimum context    71%  700 tokens
v3 + grounding rules  86%  760 tokens
Keep: v3 + grounding rules

The smallest context was not the best. v3 used a few more tokens than v2, for the grounding rules, and passed many more cases. Measuring is what showed it.

Context Limitations

Context engineering improves the model's working conditions. It does not guarantee truth, and it does not give permission to act.

Incorrect, Incomplete, and Outdated Data

If the sources are wrong, incomplete, or old, better context only makes the wrong answer easier to see. Fix the data itself.

Human Review

Some answers need a person to check them before they are used. Plan for review, and make it easy: show the sources, the confidence, and the missing information.

Legal, Medical, and Financial Decisions

Decisions about law, health, and money carry real consequences. Context can support these decisions with good evidence. It should not replace the expert who is accountable for them.

Single-Call Limits and Multi-Step Workflows

Some tasks can't be done in one model call: searching, comparing, calculating, asking follow-up questions, and acting. These need a workflow, with several steps and checks between them. The agents course covers this. Sometimes the fix is a better workflow, not a bigger context.

Accountability and Permission Limits

A model can suggest. A person or a policy must decide who is responsible. And if a user lacks permission to see something, the model must not invent it or fetch it anyway.

Where the Field Is Heading

A few trends will make context engineering more important, not less:

• Longer context windows make it possible to send more. But selection, structure, and evaluation still decide the quality.
• Multimodal inputs bring more images, audio, and video, with more uncertainty to manage.
• Persistent memory helps assistants stay consistent, but needs careful privacy and freshness rules.
• Agents and tools add more steps, and every tool result becomes part of the next step's context.
• Connector standards like MCP make outside data easy to reach. The question moves from "can the model access this?" to "should it, now, and in what shape?"

The future is not just larger context. It is better context: selected, labelled, grounded, permission-aware, and measured.

Common mistakes

Exercises

Each exercise ends with a value on the last line. Press Check to compare it with the expected answer.

What to Remember

• When an answer is wrong, check what the model saw before changing anything.
• Run the diagnostic checks in order: sources, retrieval, ranking, freshness, packing, hierarchy, tools, memory, schema, safety, and cost. Stop at the first failure.
• Traces record the final context, retrieval, tool calls, and citations. Read them first.
• Measure with an evaluation set: relevance, faithfulness, citations, schema, format, tools, media, cost, and latency.
• Improve in a loop: define, discover, split, retrieve, remember, structure, test, compare, version.
• Context engineering can't fix wrong data, and it doesn't replace human accountability.

Next: Capstone, where you take three weak assistants and turn them into strong ones, using everything in this course.