Chat Templates, Masking and Stop Tokens

By Syncfusion®, Inc. · Last updated October 28, 2026

The formatting bugs that break real fine-tunes: chat templates, loss masking, end-of-turn tokens and max sequence length, with real outputs from a tiny model.

Chat Templates, Masking and Stop Tokens

The team's first real fine-tune finishes. The loss curve looks perfect. They send it one ticket, and the answer is the right JSON, followed by a second copy, then half a third, until it hits the token limit.

Nothing was wrong with the learning rate or the data. The bug was in how the examples were formatted for the model.

Here is the course's tiny model, trained five different ways on the same data. Every output below is real.

The correct run has a validation loss of 0.066. The run without an end token has 0.085, almost the same, yet it never stops. The template-mismatch run has exactly 0.066, and its answers are garbled.

Simple lesson: a good loss curve does not prove the model works. Formatting bugs hide from it.

A Real-World Example

Think about training someone to fill in a form, where every practice form you give them has the last line torn off.

They learn the fields perfectly. But they have never seen where a form ends, so they keep writing past the bottom of the page.

Or you train them on blue forms and then hand them a green one with the boxes in different places. They knew the job, just not that layout.

Simple lesson: practise on exactly the layout you will use for real, including the ending.

What You Will Learn

In this lesson, you will learn:

1. What a chat template is, and why you should never write one by hand.
2. What loss masking is, and why you train on the answer only.
3. Why every answer needs an end-of-turn token.
4. How max_seq_length can cut answers in training.
5. Which of these bugs the loss curve can and cannot show you.

TL;DR

Chat template = the exact way a model family wraps system, user and assistant messages in special tokens. Use the tokenizer's apply_chat_template.

Loss masking = compute the loss on the assistant's answer only, not on the prompt.

End-of-turn token = the token after each answer that teaches the model to stop.

max_seq_length = the longest example, in tokens, the trainer keeps. Anything longer is cut.

Check all four with check_template.py before you train.

What the Model Really Sees

Here is the same idea in code. The template wraps each message, and only the part after assistant is trained on.

type Msg = { role: "system" | "user" | "assistant"; content: string };

// A ChatML-style template, the kind some model families use.
// Real code: tokenizer.apply_chat_template(messages). Never hand-write it.
function render(messages: Msg[]): { text: string; trainedFrom: number } {
  let text = "";
  let trainedFrom = -1;
  for (const m of messages) {
    text += `<|im_start|>${m.role}\n`;
    if (m.role === "assistant") trainedFrom = text.length;
    text += `${m.content}<|im_end|>\n`;
  }
  return { text, trainedFrom };
}

const { text, trainedFrom } = render([
  { role: "system", content: "Reply with JSON only." },
  { role: "user", content: "Cancel order 2518" },
  { role: "assistant", content: '{"intent":"cancel","order":"2518"}' },
]);
console.log(text.trimEnd());
console.log("---");
console.log("context (masked):", trainedFrom, "characters");
console.log("trained on:", JSON.stringify(text.slice(trainedFrom).trimEnd()));

using System.Text.Json;
using System.Text.Encodings.Web;

// A ChatML-style template, the kind some model families use.
// Real code: tokenizer.apply_chat_template(messages). Never hand-write it.
(string text, int trainedFrom) Render(List<Msg> messages)
{
    var text = "";
    var trainedFrom = -1;
    foreach (var m in messages)
    {
        text += $"<|im_start|>{m.Role}\n";
        if (m.Role == "assistant") trainedFrom = text.Length;
        text += $"{m.Content}<|im_end|>\n";
    }
    return (text, trainedFrom);
}

var (text, trainedFrom) = Render(new List<Msg>
{
    new("system", "Reply with JSON only."),
    new("user", "Cancel order 2518"),
    new("assistant", "{\"intent\":\"cancel\",\"order\":\"2518\"}"),
});
Console.WriteLine(text.TrimEnd());
Console.WriteLine("---");
Console.WriteLine($"context (masked): {trainedFrom} characters");
var json = new JsonSerializerOptions { Encoder = JavaScriptEncoder.UnsafeRelaxedJsonEscaping };
Console.WriteLine("trained on: " + JsonSerializer.Serialize(text[trainedFrom..].TrimEnd(), json));

record Msg(string Role, string Content);

Expected output:

<|im_start|>system
Reply with JSON only.<|im_end|>
<|im_start|>user
Cancel order 2518<|im_end|>
<|im_start|>assistant
{"intent":"cancel","order":"2518"}<|im_end|>
---
context (masked): 118 characters
trained on: "{\"intent\":\"cancel\",\"order\":\"2518\"}<|im_end|>"

The model trains on the JSON and the <|im_end|> after it. That end token is what makes generation stop.

The Four Traps

Measure Token Lengths First

Answers get cut when examples are longer than max_seq_length. The fix is to measure before training.

// Token lengths of 20 training examples (check_template.py prints these for real).
const lengths = [212, 240, 188, 301, 655, 274, 199, 530, 233, 260, 1210, 245, 318, 287, 702, 221, 264, 296, 241, 512];

for (const limit of [512, 1024, 2048]) {
  const cut = lengths.filter((n) => n > limit).length;
  console.log(`max_seq_length ${String(limit).padStart(4)}: ${cut} example(s) cut`);
}
const sorted = [...lengths].sort((a, b) => a - b);
console.log(`median ${sorted[Math.floor(sorted.length / 2)]}, longest ${sorted[sorted.length - 1]}`);

// Token lengths of 20 training examples (check_template.py prints these for real).
var lengths = new[] { 212, 240, 188, 301, 655, 274, 199, 530, 233, 260, 1210, 245, 318, 287, 702, 221, 264, 296, 241, 512 };

foreach (var limit in new[] { 512, 1024, 2048 })
{
    var cut = lengths.Count(n => n > limit);
    Console.WriteLine($"max_seq_length {limit,4}: {cut} example(s) cut");
}
var sorted = lengths.OrderBy(n => n).ToArray();
Console.WriteLine($"median {sorted[sorted.Length / 2]}, longest {sorted[^1]}");

Expected output:

max_seq_length  512: 4 example(s) cut
max_seq_length 1024: 1 example(s) cut
max_seq_length 2048: 0 example(s) cut
median 274, longest 1210

With a limit of 512, 4 of 20 examples would be cut. A cut answer teaches the model to stop mid-JSON. In the gallery, the run trained on cut answers ended with a validation loss of 2.516, because the full validation answers kept going where it had learned to stop.

python check_template.py

Common mistakes

Check Your Understanding

Exercises

What to Remember

• Use the tokenizer's chat template, the same for training and serving.
• Train on the answer only, and always include the end-of-turn token.
• Measure token lengths before setting max_seq_length.
• Two of these bugs hide from the loss curve: read real outputs.
• Run check_template.py before every first GPU run.

Next: Run a Real Fine-tune, the starter kit from data to score, on a free GPU.