# When the Fine-tune Misbehaves

Course: [Fine-tuning & Model Customization](https://www.devtutorials.ai/learn/fine-tuning) · Section: Train and Operate
Source: https://www.devtutorials.ai/learn/fine-tuning/train-and-operate/debugging-a-fine-tune

> Debug a fine-tune from its outputs: template tokens, run-on answers, cut-offs, prose and wrong values, each traced to a cause and a fix.

## When the Fine-tune Misbehaves

The fine-tune passes the ship rule: 96 of 100 eval tickets exactly right. The team is about to move on, when someone asks: "What went wrong with the other 4?"

They read them. Each one is a different bug.

None of these show up in a loss curve. You find them by reading outputs, sorting them by failure type, and tracing each type to its cause.

**Simple lesson:** the failures are a to-do list. Read every one while there are few.

### A Real-World Example

Think about a car that passes its inspection but makes a strange noise on the way home.

The inspection measured what it measures. The noise is a clue to something it did not check. A good mechanic does not ignore it because the paperwork says "pass".

**Simple lesson:** a passing score with odd failures still deserves a look.

## What You Will Learn

In this lesson, you will learn:

1. The common **output failures** of fine-tuned models, and what each one looks like.
2. How to **sort failures by type** instead of counting pass and fail.
3. How to trace each type to a **cause**: data, training format, or serving.
4. How to write an **output check** that catches malformed answers.
5. What to do when the **loss looks fine but the outputs do not**.

**Before you start**

- The predictions here are the starter kit's fixtures: base model and fine-tune on the same 100 eval tickets, with planted failures. `score.py` prints the same failure types.
- The Chat Templates lesson showed the training-side causes. This lesson starts from the outputs.

## TL;DR

Sort failures by type, then fix the cause:

- **Template tokens** in the output: serving template or stop token.
- **Never stops**: no end token in training.
- **Extra text** after the JSON: a few training answers had it, or a chat habit came back.
- **Cut off**: max_new_tokens or max_seq_length too small.
- **Prose** instead of JSON: is the adapter really loaded?
- **Wrong values**: data or labels. Read 20 and fix the data.

## Count by Type, Not Just Pass and Fail

The base model fails **24 of 100**, mostly by answering in prose (**11** cases). The fine-tune fails **4 of 100**, each in a different way.

The fine-tune's four are more interesting than the base model's 24. Each points to a bug that will grow with more traffic.

## Check the Shape First

Before checking whether a ticket's values are right, check that the output is **exactly one JSON object and nothing else**.

```ts runnable title="A first check on every output"
// A first check on every model output: exactly one JSON object, nothing else.
function problem(raw: string): string {
  const text = raw.trim();
  if (/<\|[a-z_]+\|>/.test(text)) return "template tokens";
  if (text.startsWith("```")) return "code fence";
  if (!text.startsWith("{")) return "prose";
  const end = text.lastIndexOf("}");
  try { JSON.parse(text.slice(0, text.indexOf("}") + 1)); } catch { return "invalid or cut off"; }
  return end === text.length - 1 && text.indexOf("}") === end ? "" : "text after the JSON";
}

const outputs = [
  '{"intent":"cancel","order":"2518"}',
  '{"intent":"cancel","order":"2518"}<|im_end|>',
  'Sure! I have cancelled it.',
  '{"intent":"cancel","ord',
  '{"intent":"cancel","order":"2518"}\nAnything else?',
  '```json\n{"intent":"cancel"}\n```',
];
for (const o of outputs) console.log(`${(problem(o) || "ok").padEnd(20)} ${JSON.stringify(o).slice(0, 44)}`);
```

```csharp runnable title="A first check on every output"
using System.Text.Json;
using System.Text.RegularExpressions;

// A first check on every model output: exactly one JSON object, nothing else.
string Problem(string raw)
{
    var text = raw.Trim();
    if (Regex.IsMatch(text, @"<\|[a-z_]+\|>")) return "template tokens";
    if (text.StartsWith("```")) return "code fence";
    if (!text.StartsWith("{")) return "prose";
    var end = text.LastIndexOf('}');
    try { JsonDocument.Parse(text.Substring(0, text.IndexOf('}') + 1)); } catch { return "invalid or cut off"; }
    return end == text.Length - 1 && text.IndexOf('}') == end ? "" : "text after the JSON";
}

var jsonOptions = new JsonSerializerOptions { Encoder = System.Text.Encodings.Web.JavaScriptEncoder.UnsafeRelaxedJsonEscaping };
string[] outputs = [
    "{\"intent\":\"cancel\",\"order\":\"2518\"}",
    "{\"intent\":\"cancel\",\"order\":\"2518\"}<|im_end|>",
    "Sure! I have cancelled it.",
    "{\"intent\":\"cancel\",\"ord",
    "{\"intent\":\"cancel\",\"order\":\"2518\"}\nAnything else?",
    "```json\n{\"intent\":\"cancel\"}\n```",
];
foreach (var o in outputs)
{
    var p = Problem(o);
    var json = JsonSerializer.Serialize(o, jsonOptions);
    Console.WriteLine($"{(p == "" ? "ok" : p).PadRight(20)} {json.Substring(0, Math.Min(44, json.Length))}");
}
```

Expected output:

```text
ok                   "{\"intent\":\"cancel\",\"order\":\"2518\"}"
template tokens      "{\"intent\":\"cancel\",\"order\":\"2518\"}<
prose                "Sure! I have cancelled it."
invalid or cut off   "{\"intent\":\"cancel\",\"ord"
text after the JSON  "{\"intent\":\"cancel\",\"order\":\"2518\"}\
code fence           "```json\n{\"intent\":\"cancel\"}\n```"
```

This check uses the first closing brace, which is fine for the flat ticket JSON. `score.py` in the kit parses properly, so nested objects work too.

## Spot Run-on Answers

A model that never stops often repeats itself. A cheap check: does any short piece of text appear three or more times?

```ts runnable title="Detect repetition"
// Spot a run-on answer: does any 8-character piece repeat 3+ times?
function repeats(text: string, size = 8): number {
  const seen = new Map<string, number>();
  for (let i = 0; i + size <= text.length; i++) {
    const piece = text.slice(i, i + size);
    seen.set(piece, (seen.get(piece) ?? 0) + 1);
  }
  return Math.max(0, ...seen.values());
}

const answers = [
  '{"intent":"cancel","order":"2518","priority":"high"}',
  '{"intent":"refund"}"}"}"}"}"}"}"}"}"}"}"}"}',
  '{"intent":"where"}{"intent":"where"}{"intent":"where"}',
];
for (const a of answers) console.log(`max repeats ${repeats(a)}  ${repeats(a) >= 3 ? "RUN-ON" : "ok"}`);
```

```csharp runnable title="Detect repetition"
// Spot a run-on answer: does any 8-character piece repeat 3+ times?
int Repeats(string text, int size = 8)
{
    var seen = new Dictionary<string, int>();
    for (var i = 0; i + size <= text.Length; i++)
    {
        var piece = text.Substring(i, size);
        seen[piece] = seen.GetValueOrDefault(piece) + 1;
    }
    return seen.Values.Prepend(0).Max();
}

string[] answers = [
    "{\"intent\":\"cancel\",\"order\":\"2518\",\"priority\":\"high\"}",
    "{\"intent\":\"refund\"}\"}\"}\"}\"}\"}\"}\"}\"}\"}\"}\"}\"}",
    "{\"intent\":\"where\"}{\"intent\":\"where\"}{\"intent\":\"where\"}",
];
foreach (var a in answers) Console.WriteLine($"max repeats {Repeats(a)}  {(Repeats(a) >= 3 ? "RUN-ON" : "ok")}");
```

Expected output:

```text
max repeats 1  ok
max repeats 10  RUN-ON
max repeats 3  RUN-ON
```

## From Failure to Cause

| You see | Look first at | Fix |
|---|---|---|
| Template tokens (such as `im_end` or `[INST]`) in the answer | The serving code that builds the prompt | Build the prompt with the chat template and stop on the end-of-turn token |
| JSON, then more JSON or junk | The training answers' last token | Add the end-of-turn token; check with `check_template.py`; retrain |
| JSON, then "Let me know..." | Training answers with a friendly sentence | Find and fix those rows; add a stop sequence |
| JSON cut in the middle | `max_new_tokens`, then `max_seq_length` | Raise the limit that is too small |
| A chatty sentence, no JSON | Whether the adapter is loaded at all | Log the adapter version with every answer |
| Right shape, wrong value | The labels in training | Read 20 cases; fix labels (Data Quality lesson) |

**Low loss plus broken outputs means formatting or serving.**

If the curves are healthy but the answers are malformed, do not train longer. The model learned what you gave it. Look at the template, the end token, the length limits, and the code that serves it.

- **What does the failure look like?** Ask for three raw outputs, not a summary.
- **Which versions** (base, adapter, dataset, serving code) produced them?
- **Did it pass the eval?** If yes, add these cases to the eval so it cannot pass again.
- **Is it safe to leave on?** If not, roll back first.

**Do's and Don'ts**

- **Do** count failures by type.
- **Do** read every failure when there are only a few.
- **Do** add every production failure to the eval set.
- **Do** log the adapter version with every answer.
- **Don't** train longer to fix a formatting bug.
- **Don't** hide failures with a lenient parser without fixing the cause.
- **Don't** call a 96% pass "done" before reading the other 4%.

## Common mistakes

| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| Only pass/fail counts | "96% passed, good enough." | Sort the 4% by type. | A serving bug ships and grows with traffic. |
| Lenient parsing | "Strip the junk after the JSON." | Fix why the junk is there. | The real bug stays, and gets worse on new inputs. |
| Training longer | "More epochs will fix it." | Check template, end token, limits. | Hours of GPU time for the same bug. |
| No version logging | "It must be the new adapter." | Log base, adapter, dataset, code versions. | You debug the wrong model. |
| Failures not added to the eval | "We fixed it this time." | Every failure becomes an eval case. | The same bug comes back next release. |

## Check Your Understanding

## Exercises

## What to Remember

- **Read the failures**, sorted by **type**, especially when there are few.
- Template tokens, run-ons and cut-offs are **formatting or serving** bugs, not training bugs.
- **Check the shape** of every output before checking its values.
- **Low loss + broken outputs**: check template, end token, limits and serving code.
- Every production failure becomes an **eval case**.

Next: **Evaluate Before and After**, how to prove a fine-tune helped, and catch what it broke.