# Statistics for Evals

Course: [AI Evals & Observability](https://www.devtutorials.ai/learn/evals-observability) · Section: Offline Evals
Source: https://www.devtutorials.ai/learn/evals-observability/offline-evals/statistics

> v3 scores 1.5 points higher than v1. Is that real, or luck? Learn bootstrap intervals, paired comparisons and sample size, so you can make an honest ship or no-ship call.

## Statistics for Evals

The team made a small fix to the Shop Helper prompt. They call it **v3**.

They run the 200-case eval. v1 passes **184 of 200**. v3 passes **187 of 200**.

The release notes say "v3 is **+1.5 points** better". Everyone is pleased.

Watch what is hiding behind that number.

v3 fixed **15 chats**. It also broke **12 chats** that v1 got right. The net gain is three chats.

Is that a real gain? Or did v3 just get lucky on these 200 chats?

**Simple lesson:** a score that moved a little may not have moved at all.

### A Real-World Example

Think about flipping a coin ten times.

You get six heads. Is the coin unfair?

Probably not. Flip it ten more times and you might get four.

Small samples wobble. You need many flips, or a big difference, before you can say the coin is unfair.

Eval scores work the same way. 200 chats is a small sample.

**Simple lesson:** before you trust a change, ask how much the score wobbles by luck.

## What You Will Learn

In this lesson, you will learn:

1. Why a score can change by luck alone.
2. What a **95% interval** is, and how to read one.
3. How the **bootstrap** makes an interval from your own eval results.
4. Why a **paired comparison** is the right test for two versions on the same cases.
5. How many cases you need to see a change of a given size.
6. Why an LLM judge adds its own noise, and how to shrink it.

## TL;DR

Every score has noise. A different 200 chats would give a slightly different score.

**95% interval** = a range that very likely contains the true score. If you report a score, report its interval too.

To compare two versions on the same cases, use a **paired** interval for the difference. If it crosses 0, the change is **not proven**.

Small changes need many cases. Seeing a 1-point change at a 92% pass rate takes about **11,541** cases.

## Cheat Sheet

| You see | It means | Do this |
|---|---|---|
| Paired interval above 0 | The new version is better | Ship it, if cost and speed are fine |
| Paired interval below 0 | The new version is worse | Block it |
| Paired interval crosses 0 | Not proven either way | Say "not proven", or add cases |
| A score with no interval | You cannot judge it | Ask for the interval |

## Noise: Why Scores Wobble

Your eval set is a sample of all the chats customers send.

**Sample** = a small set picked from a bigger group.

If you picked a different 200 chats, you would get a slightly different score. That wobble is called **noise**.

The model is a little random too. A tiny prompt edit can flip some chats from pass to fail, and others from fail to pass.

That is what happened with v3. 27 chats moved. Most of the moves cancel out.

**Simple lesson:** a change is only real if it is bigger than the noise.

## The Bootstrap

How big is the noise? You can measure it from your own results.

**Bootstrap** = pick cases from your eval results at random, with repeats, score them, and do it many times. The spread of those scores shows the noise.

Watch it draw by draw.

v1 scores 92%. Its 95% interval runs from **88% to 95.5%**.

That means: if we could test v1 on every possible chat, its true score is very likely somewhere in that range.

**Interval** = the range of scores the data allows. Wide interval, unsure answer. Narrow interval, sure answer.

Here is the bootstrap in a few lines of code. It uses 20 results, so you can see every part.

```ts runnable title="Bootstrap a 95% interval"
// 20 saved results for one version: true = passed the grader.
const results = [true, true, true, false, true, true, true, true, false, true,
  true, true, true, true, true, false, true, true, true, true];

// A small seeded random number generator, so every run gives the same answer.
let seed = 42;
function random(): number {
  seed = (seed * 1103515245 + 12345) % 2147483648;
  return seed / 2147483648;
}

function passRate(xs: boolean[]): number {
  return (xs.filter((x) => x).length / xs.length) * 100;
}

// The bootstrap: draw 20 results with repeats, score them, do it 1,000 times.
const scores: number[] = [];
for (let round = 0; round < 1000; round++) {
  const draw: boolean[] = [];
  for (let i = 0; i < results.length; i++) {
    draw.push(results[Math.floor(random() * results.length)]);
  }
  scores.push(passRate(draw));
}
scores.sort((a, b) => a - b);

console.log(`Score: ${passRate(results)}%`);
console.log(`95% interval: ${scores[25]}% to ${scores[974]}%`);
```

Expected output:

```text
Score: 85%
95% interval: 65% to 100%
```

With only 20 cases, the score could be anywhere from 65% to 100%. That is almost useless.

The sorted list has 1,000 scores. Dropping the lowest 25 and the highest 25 leaves the middle 95%.

**Try this:** copy the 20 results so the list has 40. Predict what happens to the interval, then run it.

## Paired: Compare on the Same Cases

v1 and v3 ran on the same 200 chats. That matters.

**Paired comparison** = compare two versions chat by chat, on the same cases, and look only at the difference.

Scored apart, v1 is **88% to 95.5%** and v3 is **89.5% to 97%**. The two ranges overlap a lot. That view throws information away.

Paired, the chats that pass on both or fail on both drop out. They tell you nothing about the change. Only the chats that moved count: **12 chats** went from pass to fail, and **15 chats** went from fail to pass.

The paired interval for "v3 minus v1" runs from **-3** to **6.5** points.

It crosses 0. So the data allows "v3 is a bit worse", "no change" and "v3 is clearly better". The honest answer is **not proven**.

Now compare v1 with v4, the bigger model. The paired interval is **+2 to +10** points. The whole range is above 0. That gain is real.

**Simple lesson:** when both versions ran on the same cases, use the paired interval. If it crosses 0, say "not proven".

## How Many Cases Do You Need?

A small change needs many cases to see. The rule of thumb below gives the number.

```ts runnable title="Cases needed to see a change"
// About how many cases you need to see a change of `points`
// when the pass rate is near `rate` (95% confidence, 80% power).
function casesNeeded(rate: number, points: number): number {
  const p = rate / 100;
  const d = points / 100;
  return Math.ceil((2 * (1.96 + 0.84) ** 2 * p * (1 - p)) / (d * d));
}

for (const points of [10, 5, 2, 1]) {
  console.log(`${points} points: ${casesNeeded(92, points)} cases`);
}
```

Expected output:

```text
10 points: 116 cases
5 points: 462 cases
2 points: 2886 cases
1 points: 11541 cases
```

At a 92% pass rate, a 200-case set can see a 10-point drop, like v2's refund problem. It cannot see a 1-point gain. That needs about **11,541** cases, and a 5-point change needs about **462**.

Halve the change you want to see, and you need about four times the cases.

**Power** = the chance your eval notices a change that is really there. 80% is a common target.

The same idea works on the Help Centre feature.

**Simple lesson:** decide the smallest change you care about first. Then size the eval set for it.

## The Judge Adds Noise Too

So far the grader was code. Code gives the same answer every time.

An LLM judge does not. It is a model, and models are a little random.

The same 200 replies were judged twice. One run passes **157 of 200**. The other passes **146 of 200**. The replies did not change.

That noise adds to the case noise. A 2-point "gain" can come from the judge alone.

Three fixes:

- Set the judge's **temperature** to 0, so it picks its most likely answer.
- **Average** several judge runs for scores that decide a release.
- Use a code check when one can catch the failure.

**Report every score with its interval.**

"v3: 93.5% (89.5% to 97%)" is a measurement. "v3: 93.5%" is a vibe with a decimal point.

**About these numbers**

- The intervals in the lab come from 1,000 bootstrap draws with a fixed seed, so they repeat exactly.
- The sample-size formula is a common rule of thumb. Real tools may use a slightly different formula, and the answer will be close.

## Where It Is Used

- **Every release decision** that compares two versions.
- **Dashboards**, so a dip of 1 point does not start a panic.
- **Planning an eval set**: how many cases you must collect.
- **Picking a model**: is the better model better by more than the noise?

**Do's and Don'ts**

- **Do** report a 95% interval with every score.
- **Do** use a paired comparison on the same cases.
- **Do** say "not proven" when the interval crosses 0.
- **Do** size the eval set for the smallest change you care about.
- **Don't** celebrate a 1-point gain on 200 cases.
- **Don't** compare scores from different eval sets as if they were paired.
- **Don't** run a judge at a high temperature for release decisions.

## Common mistakes

| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| A score with no interval | "92% is 92%." | Report the score and its 95% interval. | Nobody can tell a real change from noise. |
| Calling a small gain a win | "Up 1.5 points, ship it." | Check if the paired interval crosses 0. | The team ships, the gain vanishes, and trust in evals drops. |
| Comparing two separate ranges | "The ranges overlap, so nothing changed." | Use the paired interval for the difference. | Real gains are missed because noise from case choice is counted twice. |
| Too few cases | "50 cases is plenty." | Size the set for the change you care about. | Every result is "not proven", or worse, a lucky result is believed. |
| A hot, single-run judge | "The judge said 82%, that is the score." | Temperature 0 and average several runs. | The score swings between runs and looks like a change. |

## Check Your Understanding

## Exercises

## What to Remember

- Every eval score has **noise**. A small change can be luck.
- Report every score with its **95% interval**.
- For two versions on the same cases, use the **paired** interval of the difference.
- If the interval crosses 0, the change is **not proven**.
- Small changes need many cases. Half the change needs four times the cases.
- An LLM judge adds noise. Use temperature 0 and average several runs.

Next: **Evals in CI + v1 Export**, how to block a bad change before it ships.