Tokenization

By the Syncfusion DevTutorials team · Last updated October 20, 2026

How a tokenizer learns its pieces with byte pair encoding (BPE), how it splits new words, and how special tokens mark the chat format.

In the Tokens lesson, you used a small list of pieces that someone wrote by hand. Real tokenizers have more than 100,000 pieces. Nobody writes that list by hand.

The list is learned from text. This lesson shows the most common way to learn it, step by step, with code you can run.

The main idea: a tokenizer finds the pairs of pieces that appear together most often, and glues them into one new piece. It repeats this until it has enough pieces.

A Real-World Example

Think about how people shorten words in text messages.

At first, people typed "see you later" in full. It was so common that people started writing "see ya". Then "cya". The most common phrases got the shortest forms. Rare phrases stayed long.

Nobody decided this at a meeting. It happened because the most frequent things got squeezed first.

A tokenizer learns in the same way.

• Common phrases are pairs of pieces that appear together very often.
• Making a short form is gluing that pair into one new piece.
• Rare phrases stay long means rare words are still built from small pieces.

In simple words: the most common pieces of text get glued together first.

What You Will Learn

In this lesson, you will learn:

1. What byte pair encoding (BPE) is.
2. How to count pairs of pieces in a text.
3. How a merge glues the most common pair into one piece.
4. How repeating merges builds a vocabulary.
5. How a tokenizer splits a word it has never seen.
6. What special tokens are, and how the chat format uses them.

Byte Pair Encoding

The method most LLMs use is called byte pair encoding, or BPE. The name says what it does.

• Byte means it starts from the smallest pieces of text.
• Pair means it looks at two pieces side by side.
• Encoding means it turns text into a list of tokens.

BPE has two parts:

1. Training. Learn a list of merges from a large text. This happens once, before the model is trained.
2. Splitting. Use those merges to split any new text into tokens. This happens every time you send a message.

Training a Tiny Tokenizer

Our training text has four words. The numbers show how many times each word appears.

• low appears 5 times
• lower appears 2 times
• newest appears 6 times
• widest appears 3 times

At the start, every word is split into single letters.

Step 1: Count the Pairs

A pair is two pieces that sit next to each other. The code counts every pair, in every word, and adds the word's count each time.

Count every pair of pieces

const words = {
  "l o w": 5,
  "l o w e r": 2,
  "n e w e s t": 6,
  "w i d e s t": 3,
};

const pairs = {};
for (const word in words) {
  const pieces = word.split(" ");
  for (let i = 0; i < pieces.length - 1; i++) {
    const pair = pieces[i] + " " + pieces[i + 1];
    pairs[pair] = (pairs[pair] || 0) + words[word];
  }
}

console.log("l o:", pairs["l o"]);
console.log("e s:", pairs["e s"]);
console.log("s t:", pairs["s t"]);
console.log("w i:", pairs["w i"]);

Expected output:

l o: 7
e s: 9
s t: 9
w i: 3

l o appears in low (5 times) and lower (2 times), so it scores 7. e s appears in newest (6) and widest (3), so it scores 9.

Step 2: Find the Most Common Pair

The next step picks the pair with the highest count.

Find the most common pair

// Some of the pair counts from Step 1
const pairs = { "l o": 7, "o w": 7, "e s": 9, "s t": 9, "w e": 8, "n e": 6, "w i": 3 };

let best = "";
let bestCount = 0;
for (const pair in pairs) {
  if (pairs[pair] > bestCount) {
    best = pair;
    bestCount = pairs[pair];
  }
}

console.log("Most common pair:", best, "(" + bestCount + " times)");

Expected output:

Most common pair: e s (9 times)

e s and s t are tied at 9. The code keeps the first one it finds. Real tokenizers also need a fixed rule for ties, so the result is always the same.

Step 3: Merge the Pair

A merge glues the two pieces into one new piece, everywhere they appear side by side.

Merge e and s into es

function merge(word, a, b) {
  const pieces = word.split(" ");
  const result = [];
  for (let i = 0; i < pieces.length; i++) {
    if (pieces[i] === a && pieces[i + 1] === b) {
      result.push(a + b);
      i++;
    } else {
      result.push(pieces[i]);
    }
  }
  return result.join(" ");
}

console.log(merge("n e w e s t", "e", "s"));
console.log(merge("w i d e s t", "e", "s"));
console.log(merge("l o w", "e", "s"));

Expected output:

n e w es t
w i d es t
l o w

es is now one piece. The word low has no e s pair, so it does not change. The new piece es is added to the vocabulary.

In simple words: a merge turns two common neighbours into one piece.

Step 4: Repeat

Training is just steps 1 to 3, over and over. The next box runs four merges and prints each one.

Run four merges

let words = { "l o w": 5, "l o w e r": 2, "n e w e s t": 6, "w i d e s t": 3 };

function countPairs(words) {
  const pairs = {};
  for (const word in words) {
    const pieces = word.split(" ");
    for (let i = 0; i < pieces.length - 1; i++) {
      const pair = pieces[i] + " " + pieces[i + 1];
      pairs[pair] = (pairs[pair] || 0) + words[word];
    }
  }
  return pairs;
}

function mergeAll(words, a, b) {
  const next = {};
  for (const word in words) {
    const pieces = word.split(" ");
    const out = [];
    for (let i = 0; i < pieces.length; i++) {
      if (pieces[i] === a && pieces[i + 1] === b) { out.push(a + b); i++; }
      else out.push(pieces[i]);
    }
    next[out.join(" ")] = words[word];
  }
  return next;
}

for (let step = 1; step <= 4; step++) {
  const pairs = countPairs(words);
  let best = "";
  for (const pair in pairs) {
    if (best === "" || pairs[pair] > pairs[best]) best = pair;
  }
  const [a, b] = best.split(" ");
  words = mergeAll(words, a, b);
  console.log("Merge " + step + ": " + a + " + " + b + " -> " + a + b);
}

console.log("Words now:", Object.keys(words).join("  /  "));

Expected output:

Merge 1: e + s -> es
Merge 2: es + t -> est
Merge 3: l + o -> lo
Merge 4: lo + w -> low
Words now: low  /  low e r  /  n e w est  /  w i d est

After four merges, low is one piece and est is one piece. lower is now low plus e plus r.

The learned list of merges, in order, is: e s, es t, l o, lo w. This ordered list is the tokenizer.

Splitting a New Word

Now use the learned merges on a word the tokenizer never saw: lowest. The merges are applied in the same order they were learned.

Split a new word with the learned merges

const merges = [["e", "s"], ["es", "t"], ["l", "o"], ["lo", "w"]];

let pieces = "lowest".split("");
for (const [a, b] of merges) {
  const out = [];
  for (let i = 0; i < pieces.length; i++) {
    if (pieces[i] === a && pieces[i + 1] === b) { out.push(a + b); i++; }
    else out.push(pieces[i]);
  }
  pieces = out;
  console.log("After " + a + " + " + b + ":", pieces.join(" | "));
}

Expected output:

After e + s: l | o | w | es | t
After es + t: l | o | w | est
After l + o: lo | w | est
After lo + w: low | est

lowest was never in the training text. It still splits into two good pieces, low and est, because both parts were common.

This is why BPE works so well. Common words become one token. New words are built from common parts. And in the worst case, a word falls back to single letters or bytes, so nothing is ever impossible to write.

Byte-Level BPE

Most modern tokenizers start from bytes, not letters. A byte is a small number a computer uses to store text. Every letter, emoji, and character in every language is made of one or more bytes.

• Starting from bytes means any text can be tokenized, even emoji or rare scripts.
• Common English words are merged into one token.
• Rare characters may stay as two or three byte tokens.

Special Tokens

Some tokens are not normal text. They are special tokens, added on purpose to mark structure. Each one has its own token ID.

• End of text. Marks the end of the reply. This is the end token from the Generation Flow lesson.
• Start and end of a message. Mark where each chat message begins and ends.

You cannot type these by accident. The tokenizer adds them when it builds the chat format.

The Chat Template

In the Generation Flow lesson, you used made-up markers like <|user|>. The model you will run later in this course, Qwen2.5, uses the markers <|im_start|> and <|im_end|>. The next box builds its chat format.

Build a Qwen-style chat template

const messages = [
  { role: "system", content: "You are a helpful assistant." },
  { role: "user", content: "What is a token?" },
];

let text = "";
for (const message of messages) {
  text += "<|im_start|>" + message.role + "\n" + message.content + "<|im_end|>\n";
}
text += "<|im_start|>assistant\n";

console.log(text);

Expected output:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
What is a token?<|im_end|>
<|im_start|>assistant

The tokenizer turns <|im_start|> and <|im_end|> into one special token each. When the model writes <|im_end|> at the end of its reply, the app knows the reply is finished.

Common mistakes

Exercises

Each exercise ends with a value on the last line. Press Check to compare it with the expected answer.

What to Remember

• Real tokenizers learn their pieces from text.
• BPE means byte pair encoding: count, glue, repeat.
• A merge glues the most common pair of pieces into one new piece.
• The ordered list of merges is the tokenizer.
• New words split into learned parts, or bytes as a last resort.
• Special tokens mark structure, such as the start and end of chat messages.
• Every model has its own chat template. Use the one that comes with the model.

Next: Token Explorer, where you split real text with a real tokenizer.