Evaluate Before and After: Wins and Regressions

By Syncfusion®, Inc. · Last updated October 28, 2026

Score the base model and the fine-tune on the same cases with 95% intervals, catch catastrophic forgetting and safety regressions, and fix them before you ship.

Evaluate Before and After: Wins and Regressions

The JSON ticket fine-tune is trained. The team runs the task eval: valid JSON jumps from 79% to 97%. Ship it?

Not yet. They also run slices the fine-tune was never trained for.

The task slices clearly win. But plain questions now get a JSON ticket instead of an answer: 92% to 61%. And refusing to share another customer's data fell from 98% to 84%. That is a safety regression.

Simple lesson: measure what you trained for, and what you did not.

A Real-World Example

Think about a football player who trains only on penalty kicks for a month.

Their penalties get much better. But they stop practising running and passing, and in the next match they are slower everywhere else.

The penalty statistic says "huge improvement". The match says otherwise. You need both numbers.

Simple lesson: a narrow win can hide a wide loss.

What You Will Learn

In this lesson, you will learn:

1. Why you score the base model and the fine-tune on the same cases.
2. How to read a change with a 95% interval: win, regression or noise.
3. What catastrophic forgetting is.
4. How to fix regressions by mixing old skills back into training.
5. A ship rule that a safety regression cannot get around.

TL;DR

Score base and fine-tune on the same cases, slice by slice: task, general skills, safety.

95% interval = the range the true change probably lies in. Above zero: a win. Below zero: a regression. Crossing zero: could be noise.

Catastrophic forgetting = a fine-tune loses skills it was not trained on.

Ship only if every task slice wins and no general or safety slice regresses.

Paired Comparison

Paired = the same cases are scored on both models, so you compare case by case, not two separate averages.

// Same 20 cases scored on the base model and the fine-tune (1 = pass).
const base  = [1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,1];
const tuned = [1,0,1,0,0,1,1,0,1,1,0,1,1,0,1,1,0,1,0,1];

// A tiny seeded random generator, so the result repeats.
let seed = 7;
const rand = () => ((seed = (seed * 1103515245 + 12345) % 2147483648) / 2147483648);

const diffs = base.map((b, i) => tuned[i] - b);
const mean = (xs: number[]) => xs.reduce((s, x) => s + x, 0) / xs.length;
const means: number[] = [];
for (let r = 0; r < 2000; r++) {
  const sample = diffs.map(() => diffs[Math.floor(rand() * diffs.length)]);
  means.push(mean(sample));
}
means.sort((a, b) => a - b);
const low = means[Math.floor(2000 * 0.025)] * 100;
const high = means[Math.ceil(2000 * 0.975) - 1] * 100;
console.log(`change: ${(mean(diffs) * 100).toFixed(0)} points`);
console.log(`95% interval: ${low.toFixed(0)} to ${high.toFixed(0)}`);
console.log(high < 0 ? "regression" : low > 0 ? "win" : "could be noise");

// Same 20 cases scored on the base model and the fine-tune (1 = pass).
int[] baseScores = [1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,1];
int[] tuned      = [1,0,1,0,0,1,1,0,1,1,0,1,1,0,1,1,0,1,0,1];

// A tiny seeded random generator, so the result repeats.
long seed = 7;
double Rand() { seed = (seed * 1103515245 + 12345) % 2147483648; return seed / 2147483648.0; }

var diffs = baseScores.Select((b, i) => tuned[i] - b).ToArray();
double Mean(IEnumerable<int> xs) => xs.Average();
var means = new List<double>();
for (var r = 0; r < 2000; r++)
{
    var sample = diffs.Select(_ => diffs[(int)Math.Floor(Rand() * diffs.Length)]).ToArray();
    means.Add(Mean(sample));
}
means.Sort();
var low = means[(int)Math.Floor(2000 * 0.025)] * 100;
var high = means[(int)Math.Ceiling(2000 * 0.975) - 1] * 100;
Console.WriteLine($"change: {Mean(diffs) * 100:F0} points");
Console.WriteLine($"95% interval: {low:F0} to {high:F0}");
Console.WriteLine(high < 0 ? "regression" : low > 0 ? "win" : "could be noise");

Expected output:

change: -35 points
95% interval: -55 to -15
regression

On only 20 cases the interval is wide, but it is still entirely below zero. That is a real regression, even with few cases.

Reading the Board

On the full board the prose slice's change has an interval of -37.5 to -24.5 points. Nowhere near zero. Out of six slices, 2 regressions are real.

Summaries dropped 3 points too, but that interval crosses zero. It could be noise. Add more cases before you worry.

Fixing Forgetting

Every training example was a JSON ticket, so the model learned "always answer with a ticket". The fix is to remind it of its other skills.

After mixing in 10% of general and safety examples and retraining, prose answers are back to 88% and privacy refusals to 97%. The task win stays.

The Ship Rule

1. Every task slice is a win: its interval is above zero.
2. No general skills slice is a regression.
3. No safety slice is a regression. No exceptions, and no "we will fix it later".

Wire this into the CI gate from the Evals course, so an adapter cannot ship without passing.

type Slice = { name: string; group: "task" | "general" | "safety"; low: number; high: number };

// Each slice's 95% interval for the change, in points.
const board: Slice[] = [
  { name: "Valid JSON", group: "task", low: 12, high: 24 },
  { name: "House tone", group: "task", low: 15, high: 30.5 },
  { name: "Plain answers", group: "general", low: -37.5, high: -24.5 },
  { name: "Summaries", group: "general", low: -8.5, high: 2.5 },
  { name: "Privacy refusals", group: "safety", low: -19, high: -9 },
];

const tasksWin = board.filter((s) => s.group === "task").every((s) => s.low > 0);
const regressions = board.filter((s) => s.group !== "task" && s.high < 0).map((s) => s.name);
console.log(`task slices all win: ${tasksWin}`);
console.log(`regressions: ${regressions.join(", ") || "none"}`);
console.log(tasksWin && regressions.length === 0 ? "SHIP" : "DO NOT SHIP");

// Each slice's 95% interval for the change, in points.
List<Slice> board = [
    new("Valid JSON", "task", 12, 24),
    new("House tone", "task", 15, 30.5),
    new("Plain answers", "general", -37.5, -24.5),
    new("Summaries", "general", -8.5, 2.5),
    new("Privacy refusals", "safety", -19, -9),
];

var tasksWin = board.Where(s => s.Group == "task").All(s => s.Low > 0);
var regressions = board.Where(s => s.Group != "task" && s.High < 0).Select(s => s.Name).ToList();
Console.WriteLine($"task slices all win: {tasksWin.ToString().ToLower()}");
Console.WriteLine($"regressions: {(regressions.Count > 0 ? string.Join(", ", regressions) : "none")}");
Console.WriteLine(tasksWin && regressions.Count == 0 ? "SHIP" : "DO NOT SHIP");

record Slice(string Name, string Group, double Low, double High); // Group: "task" | "general" | "safety"

Expected output:

task slices all win: true
regressions: Plain answers, Privacy refusals
DO NOT SHIP

Run It for Real

The starter kit's score.py does the task slice of this board on real predictions, and its exit code is the gate:

python score.py --eval data/eval.jsonl --base preds/base.jsonl --tuned preds/tuned.jsonl || echo "blocked"

For general skills and safety, keep separate eval files (for example eval-general.jsonl with plain questions and eval-safety.jsonl with requests the model must refuse), generate predictions for both models, and score each slice the same way. Wire all three into the CI gate from the Evals course.

Common mistakes

Check Your Understanding

Exercises

What to Remember

• Score base and fine-tune on the same cases, by slice.
• Read the 95% interval: win, regression, or noise.
• Fine-tunes can cause catastrophic forgetting. Measure general skills and safety.
• Fix forgetting by mixing old skills back in.
• Safety regressions block release. Automate the ship rule.

Next: Distillation for Cost and Latency, how a small student model can replace a big one for 2 million tickets a month.