Reading a Training Run

By Syncfusion®, Inc. · Last updated

Diagnose six training runs from their loss curves: healthy, overfitting, stopped early, learning rate too high, and the run that looks too good. Pick the best checkpoint.

Reading a Training Run

The team ran six fine-tunes over a weekend, each with different settings and data. On Monday, six folders of logs are waiting.

Which run do they ship? The answer is in the curves.

Two runs are healthy. One overfits. One stopped too early. One has a learning rate that is far too high. And one looks perfect, which turns out to be the most worrying of all.

Simple lesson: every run tells a story. Read the curves before you read the final score.

A Real-World Example

Think about a doctor reading a heart monitor.

A steady rhythm is good. A line that jumps all over is a problem. A line that is too perfect, perfectly flat, means the sensor is not attached.

Loss curves are the same. Healthy is calm and falling. Jumpy means trouble. Too good to be true usually is.

Simple lesson: a perfect-looking result deserves the most suspicion.

What You Will Learn

In this lesson, you will learn:

  1. The five shapes a training run usually takes.
  2. How to pick the best checkpoint, not the last one.
  3. Why you check general skills and safety at each checkpoint, not just the task.
  4. What to change for each problem.
  5. Why a run that looks too good points to the data.

Before you start

  • Lesson 3 explained loss, epochs and overfitting. This lesson uses them on six full runs.
  • The runs are simulated: written to show the shapes real LoRA runs show. Open each one in the Fine-tune Lab's Run viewer.

TL;DR

  • Healthy: both lines fall and level off close together.
  • Overfitting: training loss keeps falling, validation loss turns up. Keep the checkpoint before the turn.
  • Stopped too early: both lines are still high and falling.
  • Learning rate too high: the lines jump around.
  • Too good: validation loss is almost zero. Check the data for leaks.

The Five Shapes

Diagnose five runs from their numbers

type Point = { step: number; train: number; val: number };

// Read a run the way a person would.
function diagnose(run: Point[]): string {
  const vals = run.map((p) => p.val);
  const best = vals.indexOf(Math.min(...vals));
  const last = run[run.length - 1];
  const jumps = run.slice(1).filter((p, i) => p.train - run[i].train > 0.3).length; // loss going UP
  if (jumps >= 1) return "learning rate too high: the loss jumps";
  if (last.val > vals[best] + 0.2) return `overfitting: keep step ${run[best].step}`;
  if (last.val > 1.0) return "stopped too early: loss still high";
  if (last.val < 0.2) return "suspiciously low: check for eval leaks";
  return "healthy: keep the last checkpoint";
}

const runs: Record<string, Point[]> = {
  A: [{ step: 0, train: 2.0, val: 2.0 }, { step: 100, train: 0.9, val: 1.0 }, { step: 200, train: 0.5, val: 0.6 }, { step: 300, train: 0.4, val: 0.5 }],
  B: [{ step: 0, train: 2.0, val: 2.0 }, { step: 100, train: 0.7, val: 0.8 }, { step: 200, train: 0.2, val: 0.6 }, { step: 300, train: 0.05, val: 1.1 }],
  C: [{ step: 0, train: 2.0, val: 2.0 }, { step: 100, train: 1.8, val: 1.9 }, { step: 200, train: 1.6, val: 1.7 }],
  D: [{ step: 0, train: 2.0, val: 2.0 }, { step: 100, train: 1.2, val: 1.4 }, { step: 200, train: 2.1, val: 2.2 }, { step: 300, train: 1.3, val: 1.6 }],
  E: [{ step: 0, train: 2.0, val: 2.0 }, { step: 100, train: 0.4, val: 0.4 }, { step: 200, train: 0.1, val: 0.12 }],
};
for (const [name, run] of Object.entries(runs)) console.log(`Run ${name}: ${diagnose(run)}`);

Diagnose five runs from their numbers

// Read a run the way a person would.
string Diagnose(List<Point> run)
{
    var vals = run.Select(p => p.Val).ToList();
    int best = vals.IndexOf(vals.Min());
    var last = run[run.Count - 1];
    int jumps = run.Skip(1).Where((p, i) => p.Train - run[i].Train > 0.3).Count(); // loss going UP
    if (jumps >= 1) return "learning rate too high: the loss jumps";
    if (last.Val > vals[best] + 0.2) return $"overfitting: keep step {run[best].Step}";
    if (last.Val > 1.0) return "stopped too early: loss still high";
    if (last.Val < 0.2) return "suspiciously low: check for eval leaks";
    return "healthy: keep the last checkpoint";
}

var runs = new Dictionary<string, List<Point>>
{
    ["A"] = [new(0, 2.0, 2.0), new(100, 0.9, 1.0), new(200, 0.5, 0.6), new(300, 0.4, 0.5)],
    ["B"] = [new(0, 2.0, 2.0), new(100, 0.7, 0.8), new(200, 0.2, 0.6), new(300, 0.05, 1.1)],
    ["C"] = [new(0, 2.0, 2.0), new(100, 1.8, 1.9), new(200, 1.6, 1.7)],
    ["D"] = [new(0, 2.0, 2.0), new(100, 1.2, 1.4), new(200, 2.1, 2.2), new(300, 1.3, 1.6)],
    ["E"] = [new(0, 2.0, 2.0), new(100, 0.4, 0.4), new(200, 0.1, 0.12)],
};
foreach (var (name, run) in runs) Console.WriteLine($"Run {name}: {Diagnose(run)}");

record Point(int Step, double Train, double Val);

Expected output:

Run A: healthy: keep the last checkpoint
Run B: overfitting: keep step 200
Run C: stopped too early: loss still high
Run D: learning rate too high: the loss jumps
Run E: suspiciously low: check for eval leaks

These rules are rough, but they are the same checks a person makes by eye. The cell found all five shapes.

The Best Checkpoint

Run B overfits. But it still has a good model inside it: an earlier checkpoint.

At step 300 validation loss is lowest, 0.59. By the last step it has climbed to 1.47. General skills fell from 84% to 74% along the way.

That is why you check three things at every checkpoint: the task, general skills and safety. A checkpoint that wins the task while breaking the others is not the best.

Simple lesson: save checkpoints often, then choose with the eval.

The Run That Looks Too Good

Run E finished with a validation loss of 0.11, lower than any healthy run. Everyone wants to ship it.

But Run E trained on the raw export from the Data Quality lesson, with 12 eval tickets inside it. Validation is low because the model has seen those answers.

When a run looks too good, check the data before celebrating.

Compare it with a run on cleaned data. If the "better" run used the dirtier data, the numbers are lying. The Data Quality lesson's leak check is the fix.

What to Change

ShapeWhat you seeWhat to change
HealthyBoth lines fall and level offNothing. Run the full eval board
OverfittingValidation turns upKeep the best checkpoint. Next time fewer epochs, lower rank or more data
Stopped too earlyBoth still high and fallingTrain longer, or raise the learning rate a little
Learning rate too highLines jumpLower the learning rate to about a third
Too goodValidation near zeroCheck for leaks and duplicates

What to ask when someone reports a run

  • Show me both lines, training and validation. One number is not enough.
  • Which checkpoint did you keep, and why?
  • What did general skills and safety do at that checkpoint?
  • If it looks very good: was the data checked for leaks?

Do's and Don'ts

  • Do read both loss lines for every run.
  • Do save checkpoints often and pick with the eval.
  • Do check general skills and safety at the chosen checkpoint.
  • Don't ship the last checkpoint by habit.
  • Don't trust a run that is suspiciously good without checking its data.
  • Don't change several settings between runs: you will not know which one mattered.

Common mistakes

MistakeWrong ideaCorrect ideaWhat goes wrong
Shipping the last checkpoint"Training finished, so it is done."Keep the checkpoint with the lowest validation loss.You ship the memorising version.
Only training loss"It went down, so it learned."Read validation loss too.Overfitting goes unseen.
Trusting a perfect run"Lowest loss wins."Suspiciously low loss means check the data.A leak makes you ship a weaker model.
Task score only"The task improved."Check general skills and safety per checkpoint.The model forgets things it used to do.
Many changes at once"Let's fix everything in one run."One change per run.You cannot tell what helped.

Check Your Understanding

Exercises

What to Remember

  • Read both lines: training and validation.
  • Overfitting: keep the checkpoint before validation turns up.
  • Jumpy: lower the learning rate. Still high: train longer.
  • Too good: check the data for leaks.
  • Choose checkpoints with the task, general and safety evals.

Next: When the Fine-tune Misbehaves, how to debug from the outputs when the curves look fine.