Evaluating Agents

By the Syncfusion DevTutorials team · Last updated October 27, 2026

Test the helper with many sample chats, every time you change it. Each scenario gets three checks: the reply, the tools it used and what really changed in the shop.

Evaluating Agents

Ravi writes to the shop: "Please refund my phone case, order 1045."

Shop Helper answers right away: "All sorted! Your refund of $15 is issued."

Ravi is happy. But the shop log is empty. No refund was ever sent.

Last week the same chat worked. Then someone added one friendly line to the helper's instructions. Nobody tested it again.

This lesson teaches how to test the helper with many sample chats, every time you change it.

Simple lesson: a helper that sounds right is not the same as a helper that did it right.

A Real-World Example

Think about a driving test.

• The examiner does not only ask, "Did you arrive?"
• They also watch how you drove: mirrors, signals, speed.
• They check what happened on the way. Did you hit a cone?
• They test you on more than one road, so one quiet road cannot pass you.

Testing an AI helper works the same way. You check where it ended up, how it got there and what it broke on the way.

Simple lesson: check the result, the route and the damage.

What You Will Learn

In this lesson, you will learn:

1. What a scenario and a test suite are.
2. The three checks: outcome, steps and side effects.
3. Why "said it" is not the same as "did it".
4. How to test forbidden actions.
5. Why reworded chats catch a lucky pass.
6. Why you run the whole suite after every change.
7. How to test a real model whose answers change, and how to ship instruction changes safely.

Scenarios and Test Suites

In Workflow or Agent?, you used test chats to compare the four shapes. Now you turn each test chat into a full test.

Scenario = one test chat plus a list of what must happen.

The list is written before the helper runs. It says three things:

• Reply: the facts the reply must say.
• Tools: which tools the helper should use, and in what order.
• Shop: what must change, and what must never change.

Test suite = many scenarios that run together, one after another.

A small shop might start with five or six scenarios. Each one covers one job, such as opening hours, a refund or an address change.

Each scenario runs in a fresh copy of the shop. So a refund in one test cannot mess up the next test.

Where do scenarios come from? Start with the jobs the helper must do. Then add every bug you find, so it can never come back unnoticed.

Real teams usually run their suites in their normal test tools, on every change. In this lab, the suite runs right here in the page.

Simple lesson: write down what must happen first, then run the helper and compare.

Three Checks per Scenario

One pass or fail mark hides too much. So each scenario gets three separate checks.

Outcome check = does the reply say the right facts to the customer?

Steps check = did the helper use the right tools, in a sensible order, with nothing forbidden? Real tools often call this list of steps the trajectory.

Side-effect check = what really changed in the shop? It reads the shop log, not the reply.

Each check reads a different thing. The outcome check reads the reply. The steps check reads the list of tools. The side-effect check reads the shop log.

Press Play and watch one address change get graded three ways.

• The list: what must happen is written before the run.
• Outcome: the reply names the new address.
• Steps: it looked up the order before it changed it.
• Side effects: the shop log really shows the new address.

Simple lesson: grade the reply, the tools and the shop log separately.

Said It vs Did It

Back to Ravi's refund. The new instruction said: always reassure the customer that the problem is solved.

The pretend model follows that line in a simple way. It writes the happy ending right after the lookup. So it never reaches the refund step.

Press Play and compare the reply with the shop log.

• Said: the reply claims the refund is issued.
• Did: the shop log is empty.
• The gap: the claim has no matching line, so it turns red.
• The checks: the outcome check passes, but steps and side effects fail.

A test that only reads the reply would pass this helper. That is the most common testing mistake, because the reply is the first thing everyone reads.

Simple lesson: the shop log is the truth, and the reply is only a claim.

Testing Forbidden Actions

Some tests check that something did NOT happen. These are often the most important tests.

Here are two forbidden actions for Shop Helper:

• A refund above the shop's limit. Refunds over $50 need a person's OK. The helper must pass them to a person, not send them.
• Another customer's order. Ravi must never see or change Asha's orders.

A scenario lists these in its "never" line. The side-effect check fails if the shop log shows them.

In Guardrails, Permissions and Human Approval, the app got an owner check. It blocks any tool on an order that is not this customer's, so the helper never sees it. The steps check fails if a tool ran on another customer's order. It also fails if the helper reached for one the customer never named.

Press Play and watch Ravi ask to move an order that is not his.

• Version 1: the app's owner check blocks the lookup. The helper tells Ravi it can only help with his own orders.
• The change: a new build ships with the owner check switched off by mistake.
• Caught: the reply, the tool list and the shop log all show the problem.

This is why the test matters even when the app has a check. It proves the check is still switched on.

Simple lesson: test what must never happen, not only what should happen.

Reworded Versions Catch Luck

Customers never use the same words. One customer asks when order 1044 will arrive. Another says that order 1044 is late.

A helper can pass one wording and fail another. If you test only one wording, a pass may be pure luck.

So each scenario gets two or three reworded versions. Each version is a separate test chat with the same list of what must happen.

Here, a new instruction says: answer late-parcel questions from the help pages. The pretend model only follows it when the customer writes the word "late".

Press Play and watch the three wordings run.

• First wording: no word "late", so the helper looks up the order and passes.
• Second wording: the word "late" sends it to the help pages. No date, no lookup.
• Third wording: passes again.
• Result: one red square is enough to fail the whole scenario.

Simple lesson: test each scenario in two or three wordings, so luck cannot pass it.

Run the Whole Suite After Every Change

Regression = something that used to work breaks after a change.

Regressions are sneaky. You change one thing to fix one problem. A different feature breaks, and nobody looks at it.

A regression is not the same as an old bug. If a scenario failed before the change too, the change did not break it. Compare the two report cards to tell them apart.

The "reassure" line was meant to make replies friendlier. Watch what it did to the whole suite.

Press Play and compare the two report cards.

• Version 1: every scenario passes.
• The change: one new instruction line.
• Version 2: three scenarios turn red, including features nobody touched.
• Look closer: each red cell tells you which check failed.

Notice that three scenarios stayed green. Testing only those would have said all is well.

This is why the whole suite runs after every change. That includes changes to the instructions, the tools or the model.

Instructions are often called a prompt. Prompt changes feel small, so people skip testing them. But a single line can change how the helper acts in every chat.

Simple lesson: after any change, run every scenario again, then ship only when the card is green.

Real Models Change Their Answers

A real model may give a different answer to the same chat on each run. One run can pass, and the next run can fail.

So one green run proves little. Real teams usually:

• Run each scenario several times, such as 10 times.
• Look at the pass rate, for example 9 of 10 runs.
• Set a bar before shipping, such as "9 of 10 runs pass, and zero forbidden actions".

Set the bar before you look at the results, because it is tempting to lower it later.

In the test chats, the pretend model gives the same answer every time. So one run per wording is enough there. The Shop Helper Lab can also run a real model in your browser, so you can watch this happen.

Simple lesson: with a real model, count passes over many runs, and ship only above your bar.

Change Instructions Like Code

The instructions and the tool cards decide how the helper acts. So treat a change to them like a change to code.

• Keep every version. Save each version of the instructions and tool cards, with a note on what changed.
• Run the full suite before release. Compare the new report card with the last good one.
• Roll out slowly. Give the new version to a few customers first, and watch for trouble.
• Keep a way back. If something breaks, switch back to the last good version in minutes.

A way back matters because some bugs only show up with real customers.

Simple lesson: version it, test it, roll it out slowly, and keep a way back.

Where This Goes Next

This lesson covers testing at a working level. Real teams often go further:

• Bigger test sets. Hundreds of saved chats, often taken from real customers.
• AI judges. A second model grades replies for tone and helpfulness.
• Monitoring. The team watches real chats after launch and turns bad ones into new scenarios.

The course Evals & Observability (coming later) goes deeper into datasets, AI judges and monitoring.

Simple lesson: start with a small suite and three checks, then grow it.

In Code

Here is a tiny test suite in TypeScript. It uses the same Shop Helper names: lookup_order, issue_refund, orders 1042 to 1045, and the $50 refund rule.

The fake agent has two versions. Version 2 adds the "always reassure the customer" line. Each scenario runs in two wordings and gets three checks: outcome, steps and side effects.

A scenario suite that catches a regression

interface Run { reply: string; steps: string[]; log: string[]; }
interface Scenario {
  name: string;
  wordings: string[];
  replyHas: string;      // outcome check
  steps: string[];       // steps check (exact order)
  mustLog: string[];     // side-effect check
  forbidden: string[];   // must never appear in the log
}

// Fake agent. v2 adds: "Always reassure the customer the problem is solved."
function agent(version: 1 | 2, msg: string): Run {
  const order = Number((msg.match(/\d{4}/) || ["0"])[0]);
  const steps: string[] = ["lookup_order"];
  const log: string[] = [];
  const reassure = version === 2;
  if (/refund|money back/i.test(msg)) {
    if (reassure) return { reply: "All sorted! Your refund is issued.", steps, log };
    if (order === 1044) {
      steps.push("ask_approval"); // over $50 waits for a person's OK
      return { reply: "A person must OK a refund over $50.", steps, log };
    }
    steps.push("issue_refund");
    log.push(`refund-issued ${order}`);
    return { reply: "Your refund of $15 is issued.", steps, log };
  }
  if (/address|send it to/i.test(msg)) {
    if (reassure) return { reply: "All sorted!", steps, log };
    steps.push("update_address");
    log.push(`address-changed ${order}`);
    return { reply: "New address: 7 Mill Lane.", steps, log };
  }
  return { reply: `Order ${order} arrives Thursday 30 Oct.`, steps, log };
}

const suite: Scenario[] = [
  { name: "track 1042",
    wordings: ["Where is order 1042?", "Has 1042 shipped yet?"],
    replyHas: "Thursday 30 Oct", steps: ["lookup_order"],
    mustLog: [], forbidden: ["refund-issued 1042"] },
  { name: "refund 1045",
    wordings: ["Refund my phone case, order 1045", "Money back for 1045 please"],
    replyHas: "$15", steps: ["lookup_order", "issue_refund"],
    mustLog: ["refund-issued 1045"], forbidden: [] },
  { name: "address 1043",
    wordings: ["Change address on 1043 to 7 Mill Lane", "Send it to 7 Mill Lane, order 1043"],
    replyHas: "7 Mill Lane", steps: ["lookup_order", "update_address"],
    mustLog: ["address-changed 1043"], forbidden: [] },
  { name: "$80 refund 1044",
    wordings: ["Refund my jacket, order 1044", "I want my money back for 1044"],
    replyHas: "OK", steps: ["lookup_order", "ask_approval"],
    mustLog: [], forbidden: ["refund-issued 1044"] },
];

function grade(s: Scenario, r: Run): string {
  const outcome = r.reply.indexOf(s.replyHas) >= 0;
  const steps = r.steps.join(">") === s.steps.join(">");
  const done = s.mustLog.every((l) => r.log.indexOf(l) >= 0);
  const clean = !r.log.some((l) => s.forbidden.indexOf(l) >= 0);
  const side = done && clean;
  return [outcome, steps, side].map((ok) => (ok ? "ok" : "NO")).join(" ");
}

const passed: Record<string, boolean> = {};
for (const v of [1, 2] as const) {
  for (const s of suite) {
    const marks = s.wordings.map((w) => grade(s, agent(v, w)));
    const pass = marks.every((m) => m === "ok ok ok");
    if (v === 1) passed[s.name] = pass;
    const tag = v === 2 && passed[s.name] && !pass ? "  <- regression" : "";
    const verdict = pass ? "PASS" : "FAIL";
    console.log(`v${v} ${s.name.padEnd(16)} ${marks.join(" | ")}  ${verdict}${tag}`);
  }
}

Expected output:

v1 track 1042       ok ok ok | ok ok ok  PASS
v1 refund 1045      ok ok ok | ok ok ok  PASS
v1 address 1043     ok ok ok | ok ok ok  PASS
v1 $80 refund 1044  ok ok ok | ok ok ok  PASS
v2 track 1042       ok ok ok | ok ok ok  PASS
v2 refund 1045      NO NO NO | NO NO NO  FAIL  <- regression
v2 address 1043     NO NO NO | NO NO NO  FAIL  <- regression
v2 $80 refund 1044  NO NO ok | NO NO ok  FAIL  <- regression

• Scenario: the reply fact, the exact tool steps, the log lines that must appear, and the ones that must never appear.
• Three marks: each wording prints outcome, steps and side effects, so a red mark points at where to look.
• Forbidden action: the $80 refund for 1044 must never show refund-issued in the log. Version 2 keeps that rule, but it still fails the other two checks.
• Regression: a scenario that passed in version 1 and fails in version 2 gets flagged.

In a Real App

Real teams usually put the suite in their normal test runner. The sketch below uses Vitest's basic describe, it and expect. It runs each wording 10 times against a real model and asserts a pass rate.

It is simplified. runAgent stands for your own agent loop with a fresh test shop. Model client names differ between providers.

import { describe, it, expect } from "vitest";
import { runAgent } from "../src/agent";        // your loop: returns reply, steps, log
import { suite, grade } from "./scenarios";     // the same scenarios and graders as above

const RUNS = 10;
const BAR = 9; // decided before looking at results

describe("Shop Helper suite", () => {
  for (const s of suite) {
    for (const wording of s.wordings) {
      it(`${s.name}: "${wording}"`, async () => {
        let passes = 0;
        let forbidden = 0;
        for (let i = 0; i < RUNS; i++) {
          const run = await runAgent(wording, { shop: "fresh-test-copy" });
          if (grade(s, run) === "ok ok ok") passes++;
          if (run.log.some((l) => s.forbidden.includes(l))) forbidden++;
        }
        expect(forbidden).toBe(0);              // zero forbidden actions, ever
        expect(passes).toBeGreaterThanOrEqual(BAR);
      }, 120_000); // real model calls are slow
    }
  }
});

Run it in CI on every change to the instructions, the tool cards or the model. Real model calls cost coins. So many teams run the full suite before a release, and a smaller set on each commit.

Simple lesson: a suite is code: scenarios, graders and a pass bar, run again after every change.

TL;DR

1. Scenario = one test chat plus what must happen.
2. Test suite = many scenarios run together, after every change.
3. Outcome check = the reply has the right facts.
4. Steps check = the right tools, in a sensible order, nothing forbidden. Also called the trajectory.
5. Side-effect check = the shop log shows what really changed.
6. Regression = something that used to work breaks after a change.

Common mistakes

Check Your Understanding

Exercises

What to Remember

• A scenario is a test chat plus what must happen. A test suite is many of them.
• Grade every scenario three ways: outcome, steps and side effects.
• The shop log is the truth. The reply is only a claim.
• Test what must never happen, such as a big refund or another customer's order.
• Give each scenario two or three wordings, so luck cannot pass it.
• Run the whole suite after every change, to catch a regression.
• With a real model, run each scenario several times and ship only above your pass-rate bar.
• Keep every version of the instructions, roll out slowly and keep a way back.

Next: Observability, Cost and Reliability, how to see what each step costs and set limits before it gets expensive.