# Chat Templates, Masking and Stop Tokens

Course: [Fine-tuning & Model Customization](https://www.devtutorials.ai/learn/fine-tuning) · Section: Train and Operate
Source: https://www.devtutorials.ai/learn/fine-tuning/train-and-operate/chat-templates-and-stop-tokens

> The formatting bugs that break real fine-tunes: chat templates, loss masking, end-of-turn tokens and max sequence length, with real outputs from a tiny model.

## Chat Templates, Masking and Stop Tokens

The team's first real fine-tune finishes. The loss curve looks perfect. They send it one ticket, and the answer is the right JSON, followed by a second copy, then half a third, until it hits the token limit.

Nothing was wrong with the learning rate or the data. The bug was in how the examples were **formatted** for the model.

Here is the course's tiny model, trained five different ways on the same data. Every output below is real.

The correct run has a validation loss of **0.066**. The run without an end token has **0.085**, almost the same, yet it never stops. The template-mismatch run has exactly **0.066**, and its answers are garbled.

**Simple lesson:** a good loss curve does not prove the model works. Formatting bugs hide from it.

### A Real-World Example

Think about training someone to fill in a form, where every practice form you give them has the last line torn off.

They learn the fields perfectly. But they have never seen where a form ends, so they keep writing past the bottom of the page.

Or you train them on blue forms and then hand them a green one with the boxes in different places. They knew the job, just not that layout.

**Simple lesson:** practise on exactly the layout you will use for real, including the ending.

## What You Will Learn

In this lesson, you will learn:

1. What a **chat template** is, and why you should never write one by hand.
2. What **loss masking** is, and why you train on the answer only.
3. Why every answer needs an **end-of-turn token**.
4. How **max_seq_length** can cut answers in training.
5. Which of these bugs the **loss curve can and cannot** show you.

**Before you start**

- **Token** = a small piece of text the model reads and writes. Special tokens like `<|im_end|>` mark structure, such as the end of a turn.
- The outputs in this lesson come from the course's **tiny model**, trained each wrong way by a build script. You can repeat every one in the Fine-tune Lab's **Live fine-tune** tab with the **Formatting trap** setting.

## TL;DR

**Chat template** = the exact way a model family wraps system, user and assistant messages in special tokens. Use the tokenizer's `apply_chat_template`.

**Loss masking** = compute the loss on the assistant's answer only, not on the prompt.

**End-of-turn token** = the token after each answer that teaches the model to stop.

**max_seq_length** = the longest example, in tokens, the trainer keeps. Anything longer is cut.

Check all four with `check_template.py` before you train.

## What the Model Really Sees

Here is the same idea in code. The template wraps each message, and only the part after `assistant` is trained on.

```ts runnable title="Render a chat template and find the trained part"
type Msg = { role: "system" | "user" | "assistant"; content: string };

// A ChatML-style template, the kind some model families use.
// Real code: tokenizer.apply_chat_template(messages). Never hand-write it.
function render(messages: Msg[]): { text: string; trainedFrom: number } {
  let text = "";
  let trainedFrom = -1;
  for (const m of messages) {
    text += `<|im_start|>${m.role}\n`;
    if (m.role === "assistant") trainedFrom = text.length;
    text += `${m.content}<|im_end|>\n`;
  }
  return { text, trainedFrom };
}

const { text, trainedFrom } = render([
  { role: "system", content: "Reply with JSON only." },
  { role: "user", content: "Cancel order 2518" },
  { role: "assistant", content: '{"intent":"cancel","order":"2518"}' },
]);
console.log(text.trimEnd());
console.log("---");
console.log("context (masked):", trainedFrom, "characters");
console.log("trained on:", JSON.stringify(text.slice(trainedFrom).trimEnd()));
```

```csharp runnable title="Render a chat template and find the trained part"
using System.Text.Json;
using System.Text.Encodings.Web;

// A ChatML-style template, the kind some model families use.
// Real code: tokenizer.apply_chat_template(messages). Never hand-write it.
(string text, int trainedFrom) Render(List<Msg> messages)
{
    var text = "";
    var trainedFrom = -1;
    foreach (var m in messages)
    {
        text += $"<|im_start|>{m.Role}\n";
        if (m.Role == "assistant") trainedFrom = text.Length;
        text += $"{m.Content}<|im_end|>\n";
    }
    return (text, trainedFrom);
}

var (text, trainedFrom) = Render(new List<Msg>
{
    new("system", "Reply with JSON only."),
    new("user", "Cancel order 2518"),
    new("assistant", "{\"intent\":\"cancel\",\"order\":\"2518\"}"),
});
Console.WriteLine(text.TrimEnd());
Console.WriteLine("---");
Console.WriteLine($"context (masked): {trainedFrom} characters");
var json = new JsonSerializerOptions { Encoder = JavaScriptEncoder.UnsafeRelaxedJsonEscaping };
Console.WriteLine("trained on: " + JsonSerializer.Serialize(text[trainedFrom..].TrimEnd(), json));

record Msg(string Role, string Content);
```

Expected output:

```text
<|im_start|>system
Reply with JSON only.<|im_end|>
<|im_start|>user
Cancel order 2518<|im_end|>
<|im_start|>assistant
{"intent":"cancel","order":"2518"}<|im_end|>
---
context (masked): 118 characters
trained on: "{\"intent\":\"cancel\",\"order\":\"2518\"}<|im_end|>"
```

The model trains on the JSON **and** the `<|im_end|>` after it. That end token is what makes generation stop.

- Many trainers accept examples as a **prompt** (the messages before the answer) and a **completion** (the answer).
- With that format the loss is computed on the completion only, which is the masking you want.
- The starter kit's `train_qlora.py` converts each chat line to prompt/completion for exactly this reason.

## The Four Traps

| Trap | What you see | Loss curve warns you? | Fix |
|---|---|---|---|
| No end token | Correct JSON, then it keeps going | No | Template adds the end-of-turn token after every answer |
| No loss masking | Wandering answers, prompt pieces in the output | Yes, loss is high | Train on the completion only |
| Template mismatch | Garbled answers | No | Serve with the tokenizer's chat template |
| max_seq_length too short | Every answer stops early | Yes, loss is high | Measure token lengths, raise the limit |

## Measure Token Lengths First

Answers get cut when examples are longer than `max_seq_length`. The fix is to measure before training.

```ts runnable title="How many examples a limit would cut"
// Token lengths of 20 training examples (check_template.py prints these for real).
const lengths = [212, 240, 188, 301, 655, 274, 199, 530, 233, 260, 1210, 245, 318, 287, 702, 221, 264, 296, 241, 512];

for (const limit of [512, 1024, 2048]) {
  const cut = lengths.filter((n) => n > limit).length;
  console.log(`max_seq_length ${String(limit).padStart(4)}: ${cut} example(s) cut`);
}
const sorted = [...lengths].sort((a, b) => a - b);
console.log(`median ${sorted[Math.floor(sorted.length / 2)]}, longest ${sorted[sorted.length - 1]}`);
```

```csharp runnable title="How many examples a limit would cut"
// Token lengths of 20 training examples (check_template.py prints these for real).
var lengths = new[] { 212, 240, 188, 301, 655, 274, 199, 530, 233, 260, 1210, 245, 318, 287, 702, 221, 264, 296, 241, 512 };

foreach (var limit in new[] { 512, 1024, 2048 })
{
    var cut = lengths.Count(n => n > limit);
    Console.WriteLine($"max_seq_length {limit,4}: {cut} example(s) cut");
}
var sorted = lengths.OrderBy(n => n).ToArray();
Console.WriteLine($"median {sorted[sorted.Length / 2]}, longest {sorted[^1]}");
```

Expected output:

```text
max_seq_length  512: 4 example(s) cut
max_seq_length 1024: 1 example(s) cut
max_seq_length 2048: 0 example(s) cut
median 274, longest 1210
```

With a limit of 512, **4 of 20** examples would be cut. A cut answer teaches the model to stop mid-JSON. In the gallery, the run trained on cut answers ended with a validation loss of 2.516, because the full validation answers kept going where it had learned to stop.

**Run check_template.py before every first run.**

It prints one example exactly as the model sees it, shows the token after the answer, checks that the serving prompt is a prefix of the training text, and counts examples over the limit. Ten seconds of checking saves a wasted GPU run.

```bash
python check_template.py
```

- **Did someone print one formatted example** and look at it?
- **Is the loss on the answer only?**
- **Does every answer end with the end-of-turn token?**
- **How many examples exceed max_seq_length?** The answer should be zero, or a known few.
- **Is the serving code using the same template?** Ask to see the line that builds the prompt.

**Do's and Don'ts**

- **Do** use the tokenizer's chat template for training and serving.
- **Do** train on the answer only.
- **Do** check the end-of-turn token after every answer.
- **Do** measure token lengths before choosing max_seq_length.
- **Don't** hand-write templates with string concatenation.
- **Don't** trust a low loss as proof the outputs are right.
- **Don't** change the serving prompt without re-running the eval.

## Common mistakes

| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| Hand-written template | "It is just some tags around the text." | Use the tokenizer's chat template. | Training and serving disagree by one token, and outputs break. |
| No end token | "The model knows when the JSON closes." | Train the end-of-turn token after each answer. | The model writes until max_new_tokens runs out. |
| Loss on the prompt | "More tokens to learn from." | Mask the prompt; learn the answer. | Capacity wasted on copying customer messages. |
| Guessed max length | "512 is a common default." | Measure lengths first. | Long examples are cut and teach early stopping. |
| Trusting the loss | "Loss is 0.07, ship it." | Read real outputs too. | A model that never stops reaches production. |

## Check Your Understanding

## Exercises

## What to Remember

- Use the tokenizer's **chat template**, the same for training and serving.
- Train on the **answer only**, and always include the **end-of-turn token**.
- **Measure token lengths** before setting max_seq_length.
- Two of these bugs **hide from the loss curve**: read real outputs.
- Run **check_template.py** before every first GPU run.

Next: **Run a Real Fine-tune**, the starter kit from data to score, on a free GPU.