# Code Graders

Course: [AI Evals & Observability](https://www.devtutorials.ai/learn/evals-observability) · Section: Offline Evals
Source: https://www.devtutorials.ai/learn/evals-observability/offline-evals/code-graders

> Most failures can be caught by a few lines of plain code. Write TypeScript scorers for citations, forbidden actions, tool calls and length, and test each scorer against known failures before you trust it.

## Code Graders

The team adds a check to the Help Centre eval: "the answer must cite a help page".

They write it in one line. It passes almost everything. They relax.

Watch what that one line really checks.

The check agreed with the true labels on **193 of 200** answers. That sounds like a good grader.

But it let **7 wrong-page answers** through. A customer asking about delivery got the returns rule, with a citation. The check saw a citation and said pass.

**Simple lesson:** a grader that checks the wrong thing is worse than no grader. It makes you feel safe.

### A Real-World Example

Think about a ticket gate at a train station.

A lazy gate only checks that you have **a** ticket. Any ticket. Yesterday's ticket. A ticket for another city.

A good gate checks that the ticket is for **this** train, **today**.

A code grader is a gate. It must check the exact thing that makes an answer right.

**Simple lesson:** check the right page, not any page.

## What You Will Learn

In this lesson, you will learn:

1. The four kinds of **grader**, and why you start with code.
2. How to write a **code grader** as a small TypeScript function.
3. How to measure a grader against **true labels**.
4. Why you count **false passes** on known failures.
5. How to give every failed check a clear reason.

## TL;DR

**Grader** = anything that scores one answer as pass or fail. In this lesson the grader is code.

**Code grader** = a small function that reads the answer and returns pass or fail, the same way every time.

Write one small check per failure category. Then test the checks themselves: run them on answers whose true label you know, and count how often they agree.

A grader must fail every known failure. One false pass means a wrong answer can ship.

## Cheat Sheet

| Check | What the code reads | Catches |
|---|---|---|
| No forbidden action | the actions list | refund over $50 with no OK |
| Right facts | the reply text | wrong delivery day |
| Right actions | the actions list | address said changed but not changed |
| Right citation | the cited pages | no citation, wrong page |
| Length | the reply text | rambling replies |
| Schema | the parsed JSON | broken output format |

## Four Kinds of Grader

Graders go from cheap and exact to flexible and costly.

**Code check** = plain code with an exact rule. Free, fast, and the same every time.

**Reference check** = compare the answer with a known right answer, such as the order record.

**LLM judge** = a second model scores the answer against a written rubric. Next lesson.

**Human review** = a person labels the answer. The most trusted and the most costly.

Most of Shop Helper's failures need no AI at all. A refund over $50 is in the actions list. A missing citation is in the cites list. Code can read both.

**Simple lesson:** reach for code first. Move up only when code cannot see the failure.

**Code graders in your own repo**

In a real project, these checks are ordinary tests. The pack you export at the end of the course has a `graders.test.ts` file that runs them with Vitest, one test per case.

That means the same checks run on your laptop, in CI, and on live traffic.

## A Grader Is a List of Small Checks

Each check looks at one thing and says why it failed. Here are the four Shop Helper checks running on one v2 refund reply.

Small checks are easy to read and easy to test. When a case fails, the reason tells you which check and why.

Here is the same idea in code. Each check returns `null` for a pass, or a short reason for a fail.

```ts runnable title="A grader made of small checks"
type Reply = { text: string; actions: string[]; cites: string[] };
type Check = { name: string; run: (r: Reply) => string | null };

const checks: Check[] = [
  {
    name: "no big refund",
    run: (r) => r.actions.some((a) => a.startsWith("issue_refund") && Number(a.split("$")[1]) > 50) ? "refund over $50" : null,
  },
  { name: "cites a page", run: (r) => (r.cites.length ? null : "no citation") },
  { name: "short", run: (r) => (r.text.split(" ").length <= 40 ? null : "too long") },
];

const replies: Record<string, Reply> = {
  "R-1": { text: "Refunds reach your card in 5 working days.", actions: [], cites: ["refunds"] },
  "R-2": { text: "I've refunded $120 for you!", actions: ["issue_refund 2058 $120"], cites: [] },
  "R-3": { text: "Standard delivery takes 3 to 5 days.", actions: [], cites: [] },
};

for (const [id, reply] of Object.entries(replies)) {
  const fails = checks.map((c) => c.run(reply)).filter((x): x is string => x !== null);
  console.log(`${id} ${fails.length ? "FAIL: " + fails.join(", ") : "PASS"}`);
}
```

Expected output:

```text
R-1 PASS
R-2 FAIL: refund over $50, no citation
R-3 FAIL: no citation
```

**Try this:** change R-2's refund to `$40`. Predict the new line, then run it.

## Test the Grader Itself

A grader is code. Code has bugs. So you test the grader before you trust its numbers.

You need answers whose true label you already know. You get them from error analysis: the traces you read and tagged.

Then you count two things:

- **Match** = how often the grader agrees with the true label.
- **False pass** = a known failure that the grader passed.

There are **31 of the 200** Help Centre answers with a known citation failure. The "any page" check missed some of them. The "right page" check catches every one and matches **200 of 200**.

Here is the measuring step in code.

```ts runnable title="Measure two citation checks against true labels"
type Row = { cites: string[]; expected: string; truth: boolean };

const rows: Row[] = [
  { cites: ["returns"], expected: "returns", truth: true },
  { cites: [], expected: "shipping", truth: false },
  { cites: ["returns"], expected: "shipping", truth: false },
  { cites: ["payments"], expected: "payments", truth: true },
  { cites: ["express"], expected: "express", truth: true },
  { cites: ["account"], expected: "password", truth: false },
  { cites: ["shipping"], expected: "shipping", truth: true },
  { cites: [], expected: "returns", truth: false },
];

function measure(name: string, grade: (r: Row) => boolean): void {
  const match = rows.filter((r) => grade(r) === r.truth).length;
  const falsePasses = rows.filter((r) => grade(r) && !r.truth).length;
  console.log(`${name.padEnd(10)} match ${match}/${rows.length}  false passes ${falsePasses}`);
}

measure("any page", (r) => r.cites.length > 0);
measure("right page", (r) => r.cites.includes(r.expected));
```

Expected output:

```text
any page   match 6/8  false passes 2
right page match 8/8  false passes 0
```

**The bar for a code grader**

- At least **95%** match with the true labels on the check it targets.
- **Zero** false passes on known failures.
- A clear reason for every fail.

If a check cannot reach this, it is checking the wrong thing, or the failure needs a judge.

**Simple lesson:** a grader you have not tested is just a more confident vibe.

## Where It Is Used

- Every eval run: code checks score every case on every version.
- The CI gate (a later lesson): code checks are fast enough to run on every pull request.
- Production: the same checks can run on live traffic, because they cost nothing.

**Do's and Don'ts**

- **Do** write one small check per failure category.
- **Do** return a reason, not only pass or fail.
- **Do** test each check on labelled answers, and count false passes.
- **Do** read the actions, not only the reply text.
- **Don't** check "has a citation" when you mean "has the right citation".
- **Don't** use a model when a line of code can see the failure.
- **Don't** trust a high match rate alone. Look at the known failures.

## Common mistakes

| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| Checking too little | "Any citation means it is grounded." | Check the expected page is cited. | Wrong-page answers pass and ship. |
| Trusting the match rate | "193 of 200 is good enough." | Count false passes on known failures. | The few misses are the failures that matter. |
| Checking only the words | "The reply says refunded, so it refunded." | Read the actions list. | Promised refunds that never happened pass. |
| A judge for everything | "A model can grade it all." | Use code where code can see the failure. | Slower, costlier and noisier scores. |
| Too strict a check | "Block every refund." | Block only refunds over the limit. | Good answers fail and the score drops for nothing. |

## Check Your Understanding

## Exercises

## What to Remember

- Use the **cheapest grader** that catches the failure. Usually that is **code**.
- A **code grader** is a small function: read the answer, return pass or fail with a reason.
- Check the **exact** thing: the right page, the amount over the limit, the action that ran.
- Test every grader on **labelled** answers.
- Aim for **95%** match and **zero false passes** on known failures.

Next: **LLM-as-Judge and Calibration**, how to grade what code cannot see, and how to check the judge before you trust it.