A citation check matches 193 of 200 labels but passes 7 known failures. Is it good enough?

No, it must fail every known failure

How can code catch a refund over $50?

By reading the actions list and the amount

Where do the true labels for testing a grader come from?

Error analysis, where people read and tagged traces

Code Graders

By the Syncfusion DevTutorials team · Last updated October 27, 2026

Most failures can be caught by a few lines of plain code. Write TypeScript scorers for citations, forbidden actions, tool calls and length, and test each scorer against known failures before you trust it.

Code Graders

The team adds a check to the Help Centre eval: "the answer must cite a help page".

They write it in one line. It passes almost everything. They relax.

Watch what that one line really checks.

The check agreed with the true labels on 193 of 200 answers. That sounds like a good grader.

But it let 7 wrong-page answers through. A customer asking about delivery got the returns rule, with a citation. The check saw a citation and said pass.

Simple lesson: a grader that checks the wrong thing is worse than no grader. It makes you feel safe.

A Real-World Example

Think about a ticket gate at a train station.

A lazy gate only checks that you have a ticket. Any ticket. Yesterday's ticket. A ticket for another city.

A good gate checks that the ticket is for this train, today.

A code grader is a gate. It must check the exact thing that makes an answer right.

Simple lesson: check the right page, not any page.

What You Will Learn

In this lesson, you will learn:

The four kinds of grader, and why you start with code.
How to write a code grader as a small TypeScript function.
How to measure a grader against true labels.
Why you count false passes on known failures.
How to give every failed check a clear reason.

TL;DR

Grader = anything that scores one answer as pass or fail. In this lesson the grader is code.

Code grader = a small function that reads the answer and returns pass or fail, the same way every time.

Write one small check per failure category. Then test the checks themselves: run them on answers whose true label you know, and count how often they agree.

A grader must fail every known failure. One false pass means a wrong answer can ship.

Cheat Sheet

Check	What the code reads	Catches
No forbidden action	the actions list	refund over $50 with no OK
Right facts	the reply text	wrong delivery day
Right actions	the actions list	address said changed but not changed
Right citation	the cited pages	no citation, wrong page
Length	the reply text	rambling replies
Schema	the parsed JSON	broken output format

Four Kinds of Grader

Graders go from cheap and exact to flexible and costly.

Four kinds of grader. Use the cheapest one that catches the failure.

Code check = plain code with an exact rule. Free, fast, and the same every time.

Reference check = compare the answer with a known right answer, such as the order record.

LLM judge = a second model scores the answer against a written rubric. Next lesson.

Human review = a person labels the answer. The most trusted and the most costly.

Most of Shop Helper's failures need no AI at all. A refund over $50 is in the actions list. A missing citation is in the cites list. Code can read both.

Simple lesson: reach for code first. Move up only when code cannot see the failure.

Code graders in your own repo

In a real project, these checks are ordinary tests. The pack you export at the end of the course has a graders.test.ts file that runs them with Vitest, one test per case.

That means the same checks run on your laptop, in CI, and on live traffic.

A Grader Is a List of Small Checks

Each check looks at one thing and says why it failed. Here are the four Shop Helper checks running on one v2 refund reply.

Small checks are easy to read and easy to test. When a case fails, the reason tells you which check and why.

Here is the same idea in code. Each check returns null for a pass, or a short reason for a fail.

A grader made of small checks

type Reply = { text: string; actions: string[]; cites: string[] };
type Check = { name: string; run: (r: Reply) => string | null };

const checks: Check[] = [
  {
    name: "no big refund",
    run: (r) => r.actions.some((a) => a.startsWith("issue_refund") && Number(a.split("$")[1]) > 50) ? "refund over $50" : null,
  },
  { name: "cites a page", run: (r) => (r.cites.length ? null : "no citation") },
  { name: "short", run: (r) => (r.text.split(" ").length <= 40 ? null : "too long") },
];

const replies: Record<string, Reply> = {
  "R-1": { text: "Refunds reach your card in 5 working days.", actions: [], cites: ["refunds"] },
  "R-2": { text: "I've refunded $120 for you!", actions: ["issue_refund 2058 $120"], cites: [] },
  "R-3": { text: "Standard delivery takes 3 to 5 days.", actions: [], cites: [] },
};

for (const [id, reply] of Object.entries(replies)) {
  const fails = checks.map((c) => c.run(reply)).filter((x): x is string => x !== null);
  console.log(`${id} ${fails.length ? "FAIL: " + fails.join(", ") : "PASS"}`);
}

Expected output:

R-1 PASS
R-2 FAIL: refund over $50, no citation
R-3 FAIL: no citation

Try this: change R-2's refund to $40. Predict the new line, then run it.

Test the Grader Itself

A grader is code. Code has bugs. So you test the grader before you trust its numbers.

You need answers whose true label you already know. You get them from error analysis: the traces you read and tagged.

Then you count two things:

Match = how often the grader agrees with the true label.
False pass = a known failure that the grader passed.

There are 31 of the 200 Help Centre answers with a known citation failure. The "any page" check missed some of them. The "right page" check catches every one and matches 200 of 200.

Here is the measuring step in code.

Measure two citation checks against true labels

type Row = { cites: string[]; expected: string; truth: boolean };

const rows: Row[] = [
  { cites: ["returns"], expected: "returns", truth: true },
  { cites: [], expected: "shipping", truth: false },
  { cites: ["returns"], expected: "shipping", truth: false },
  { cites: ["payments"], expected: "payments", truth: true },
  { cites: ["express"], expected: "express", truth: true },
  { cites: ["account"], expected: "password", truth: false },
  { cites: ["shipping"], expected: "shipping", truth: true },
  { cites: [], expected: "returns", truth: false },
];

function measure(name: string, grade: (r: Row) => boolean): void {
  const match = rows.filter((r) => grade(r) === r.truth).length;
  const falsePasses = rows.filter((r) => grade(r) && !r.truth).length;
  console.log(`${name.padEnd(10)} match ${match}/${rows.length}  false passes ${falsePasses}`);
}

measure("any page", (r) => r.cites.length > 0);
measure("right page", (r) => r.cites.includes(r.expected));

Expected output:

any page   match 6/8  false passes 2
right page match 8/8  false passes 0

The bar for a code grader

At least 95% match with the true labels on the check it targets.
Zero false passes on known failures.
A clear reason for every fail.

If a check cannot reach this, it is checking the wrong thing, or the failure needs a judge.

Simple lesson: a grader you have not tested is just a more confident vibe.

Where It Is Used

Every eval run: code checks score every case on every version.
The CI gate (a later lesson): code checks are fast enough to run on every pull request.
Production: the same checks can run on live traffic, because they cost nothing.

Do's and Don'ts

Do write one small check per failure category.
Do return a reason, not only pass or fail.
Do test each check on labelled answers, and count false passes.
Do read the actions, not only the reply text.
Don't check "has a citation" when you mean "has the right citation".
Don't use a model when a line of code can see the failure.
Don't trust a high match rate alone. Look at the known failures.

Common mistakes

Mistake	Wrong idea	Correct idea	What goes wrong
Checking too little	"Any citation means it is grounded."	Check the expected page is cited.	Wrong-page answers pass and ship.
Trusting the match rate	"193 of 200 is good enough."	Count false passes on known failures.	The few misses are the failures that matter.
Checking only the words	"The reply says refunded, so it refunded."	Read the actions list.	Promised refunds that never happened pass.
A judge for everything	"A model can grade it all."	Use code where code can see the failure.	Slower, costlier and noisier scores.
Too strict a check	"Block every refund."	Block only refunds over the limit.	Good answers fail and the score drops for nothing.

Check Your Understanding

Exercises

What to Remember

Use the cheapest grader that catches the failure. Usually that is code.
A code grader is a small function: read the answer, return pass or fail with a reason.
Check the exact thing: the right page, the amount over the limit, the action that ran.
Test every grader on labelled answers.
Aim for 95% match and zero false passes on known failures.

Next: LLM-as-Judge and Calibration, how to grade what code cannot see, and how to check the judge before you trust it.