# Tokenization

Course: [Large Language Models](https://www.devtutorials.ai/learn/llm) · Section: Tokens
Source: https://www.devtutorials.ai/learn/llm/tokens/tokenization

> How a tokenizer learns its pieces with byte pair encoding (BPE), how it splits new words, and how special tokens mark the chat format.

In the Tokens lesson, you used a small list of pieces that someone wrote by hand. Real tokenizers have more than 100,000 pieces. Nobody writes that list by hand.

The list is **learned** from text. This lesson shows the most common way to learn it, step by step, with code you can run.

**The main idea:** a tokenizer finds the pairs of pieces that appear together most often, and glues them into one new piece. It repeats this until it has enough pieces.

## A Real-World Example

Think about how people shorten words in text messages.

At first, people typed "see you later" in full. It was so common that people started writing "see ya". Then "cya". The most common phrases got the shortest forms. Rare phrases stayed long.

Nobody decided this at a meeting. It happened because the most frequent things got squeezed first.

A tokenizer learns in the same way.

- **Common phrases** are pairs of pieces that appear together very often.
- **Making a short form** is gluing that pair into one new piece.
- **Rare phrases stay long** means rare words are still built from small pieces.

**In simple words:** the most common pieces of text get glued together first.

## What You Will Learn

In this lesson, you will learn:

1. What byte pair encoding (BPE) is.
2. How to count pairs of pieces in a text.
3. How a merge glues the most common pair into one piece.
4. How repeating merges builds a vocabulary.
5. How a tokenizer splits a word it has never seen.
6. What special tokens are, and how the chat format uses them.

Every code box on this page runs a real part of the BPE method on a tiny text. Real tokenizers run the same steps on billions of words and stop after about 100,000 merges or more.

## Byte Pair Encoding

The method most LLMs use is called **byte pair encoding**, or **BPE**. The name says what it does.

- **Byte** means it starts from the smallest pieces of text.
- **Pair** means it looks at two pieces side by side.
- **Encoding** means it turns text into a list of tokens.

BPE has two parts:

1. **Training.** Learn a list of merges from a large text. This happens once, before the model is trained.
2. **Splitting.** Use those merges to split any new text into tokens. This happens every time you send a message.

## Training a Tiny Tokenizer

Our training text has four words. The numbers show how many times each word appears.

- `low` appears 5 times
- `lower` appears 2 times
- `newest` appears 6 times
- `widest` appears 3 times

At the start, every word is split into single letters.

### Step 1: Count the Pairs

A **pair** is two pieces that sit next to each other. The code counts every pair, in every word, and adds the word's count each time.

```js runnable title="Count every pair of pieces"
const words = {
  "l o w": 5,
  "l o w e r": 2,
  "n e w e s t": 6,
  "w i d e s t": 3,
};

const pairs = {};
for (const word in words) {
  const pieces = word.split(" ");
  for (let i = 0; i < pieces.length - 1; i++) {
    const pair = pieces[i] + " " + pieces[i + 1];
    pairs[pair] = (pairs[pair] || 0) + words[word];
  }
}

console.log("l o:", pairs["l o"]);
console.log("e s:", pairs["e s"]);
console.log("s t:", pairs["s t"]);
console.log("w i:", pairs["w i"]);
```

Expected output:

```text
l o: 7
e s: 9
s t: 9
w i: 3
```

`l o` appears in `low` (5 times) and `lower` (2 times), so it scores 7. `e s` appears in `newest` (6) and `widest` (3), so it scores 9.

### Step 2: Find the Most Common Pair

The next step picks the pair with the highest count.

```js runnable title="Find the most common pair"
// Some of the pair counts from Step 1
const pairs = { "l o": 7, "o w": 7, "e s": 9, "s t": 9, "w e": 8, "n e": 6, "w i": 3 };

let best = "";
let bestCount = 0;
for (const pair in pairs) {
  if (pairs[pair] > bestCount) {
    best = pair;
    bestCount = pairs[pair];
  }
}

console.log("Most common pair:", best, "(" + bestCount + " times)");
```

Expected output:

```text
Most common pair: e s (9 times)
```

`e s` and `s t` are tied at 9. The code keeps the first one it finds. Real tokenizers also need a fixed rule for ties, so the result is always the same.

### Step 3: Merge the Pair

A **merge** glues the two pieces into one new piece, everywhere they appear side by side.

```js runnable title="Merge e and s into es"
function merge(word, a, b) {
  const pieces = word.split(" ");
  const result = [];
  for (let i = 0; i < pieces.length; i++) {
    if (pieces[i] === a && pieces[i + 1] === b) {
      result.push(a + b);
      i++;
    } else {
      result.push(pieces[i]);
    }
  }
  return result.join(" ");
}

console.log(merge("n e w e s t", "e", "s"));
console.log(merge("w i d e s t", "e", "s"));
console.log(merge("l o w", "e", "s"));
```

Expected output:

```text
n e w es t
w i d es t
l o w
```

`es` is now one piece. The word `low` has no `e s` pair, so it does not change. The new piece `es` is added to the vocabulary.

**In simple words:** a merge turns two common neighbours into one piece.

### Step 4: Repeat

Training is just steps 1 to 3, over and over. The next box runs four merges and prints each one.

```js runnable title="Run four merges"
let words = { "l o w": 5, "l o w e r": 2, "n e w e s t": 6, "w i d e s t": 3 };

function countPairs(words) {
  const pairs = {};
  for (const word in words) {
    const pieces = word.split(" ");
    for (let i = 0; i < pieces.length - 1; i++) {
      const pair = pieces[i] + " " + pieces[i + 1];
      pairs[pair] = (pairs[pair] || 0) + words[word];
    }
  }
  return pairs;
}

function mergeAll(words, a, b) {
  const next = {};
  for (const word in words) {
    const pieces = word.split(" ");
    const out = [];
    for (let i = 0; i < pieces.length; i++) {
      if (pieces[i] === a && pieces[i + 1] === b) { out.push(a + b); i++; }
      else out.push(pieces[i]);
    }
    next[out.join(" ")] = words[word];
  }
  return next;
}

for (let step = 1; step <= 4; step++) {
  const pairs = countPairs(words);
  let best = "";
  for (const pair in pairs) {
    if (best === "" || pairs[pair] > pairs[best]) best = pair;
  }
  const [a, b] = best.split(" ");
  words = mergeAll(words, a, b);
  console.log("Merge " + step + ": " + a + " + " + b + " -> " + a + b);
}

console.log("Words now:", Object.keys(words).join("  /  "));
```

Expected output:

```text
Merge 1: e + s -> es
Merge 2: es + t -> est
Merge 3: l + o -> lo
Merge 4: lo + w -> low
Words now: low  /  low e r  /  n e w est  /  w i d est
```

After four merges, `low` is one piece and `est` is one piece. `lower` is now `low` plus `e` plus `r`.

The learned list of merges, in order, is: `e s`, `es t`, `l o`, `lo w`. This ordered list **is** the tokenizer.

**Easy Way to Remember:** BPE is **count, glue, repeat**. Count the pairs, glue the most common one, and do it again.

## Splitting a New Word

Now use the learned merges on a word the tokenizer never saw: `lowest`. The merges are applied in the same order they were learned.

```js runnable title="Split a new word with the learned merges"
const merges = [["e", "s"], ["es", "t"], ["l", "o"], ["lo", "w"]];

let pieces = "lowest".split("");
for (const [a, b] of merges) {
  const out = [];
  for (let i = 0; i < pieces.length; i++) {
    if (pieces[i] === a && pieces[i + 1] === b) { out.push(a + b); i++; }
    else out.push(pieces[i]);
  }
  pieces = out;
  console.log("After " + a + " + " + b + ":", pieces.join(" | "));
}
```

Expected output:

```text
After e + s: l | o | w | es | t
After es + t: l | o | w | est
After l + o: lo | w | est
After lo + w: low | est
```

`lowest` was never in the training text. It still splits into two good pieces, `low` and `est`, because both parts were common.

This is why BPE works so well. Common words become one token. New words are built from common parts. And in the worst case, a word falls back to single letters or bytes, so nothing is ever impossible to write.

## Byte-Level BPE

Most modern tokenizers start from **bytes**, not letters. A byte is a small number a computer uses to store text. Every letter, emoji, and character in every language is made of one or more bytes.

- Starting from bytes means **any** text can be tokenized, even emoji or rare scripts.
- Common English words are merged into one token.
- Rare characters may stay as two or three byte tokens.

## Special Tokens

Some tokens are not normal text. They are **special tokens**, added on purpose to mark structure. Each one has its own token ID.

- **End of text.** Marks the end of the reply. This is the end token from the Generation Flow lesson.
- **Start and end of a message.** Mark where each chat message begins and ends.

You cannot type these by accident. The tokenizer adds them when it builds the chat format.

### The Chat Template

In the Generation Flow lesson, you used made-up markers like `<|user|>`. The model you will run later in this course, Qwen2.5, uses the markers `<|im_start|>` and `<|im_end|>`. The next box builds its chat format.

```js runnable title="Build a Qwen-style chat template"
const messages = [
  { role: "system", content: "You are a helpful assistant." },
  { role: "user", content: "What is a token?" },
];

let text = "";
for (const message of messages) {
  text += "<|im_start|>" + message.role + "\n" + message.content + "<|im_end|>\n";
}
text += "<|im_start|>assistant\n";

console.log(text);
```

Expected output:

```text
<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
What is a token?<|im_end|>
<|im_start|>assistant
```

The tokenizer turns `<|im_start|>` and `<|im_end|>` into one special token each. When the model writes `<|im_end|>` at the end of its reply, the app knows the reply is finished.

Every model family has its own chat template and special tokens. If you send a model the wrong template, its answers get worse or never stop. Libraries normally build the template for you. Use the one that comes with the model.

## Common mistakes

| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| Thinking tokens are chosen by hand | "Someone wrote a list of 150,000 pieces." | The pieces are learned from text by counting and merging pairs. | You expect tokens to match grammar or meaning. They follow frequency. |
| Merging in any order | "The order of merges does not matter." | Merges are applied in the order they were learned. | Text splits into different tokens than the model expects. |
| Thinking new words break the tokenizer | "A word it never saw cannot be tokenized." | New words are built from learned parts, or single bytes as a last resort. | You add extra code to handle words that already work. |
| Expecting the same splits everywhere | "Every model splits `lowest` the same way." | Each model learned its own merges from its own text. | Token counts and IDs do not match across models. |
| Using the wrong chat template | "Any message format will do." | Each model was trained with its own special tokens. | Replies get worse, or the model does not stop. |
| Treating special tokens as text | "`<\|im_end\|>` is just some letters." | It is one special token with its own ID. | Prompts that contain the marker confuse the model. |

## Exercises

Each exercise ends with a value on the last line. Press **Check** to compare it with the expected answer.

## What to Remember

- Real tokenizers **learn** their pieces from text.
- **BPE** means byte pair encoding: **count, glue, repeat**.
- A **merge** glues the most common pair of pieces into one new piece.
- The ordered list of merges **is** the tokenizer.
- New words split into **learned parts**, or bytes as a last resort.
- **Special tokens** mark structure, such as the start and end of chat messages.
- Every model has its own **chat template**. Use the one that comes with the model.

Next: **Token Explorer**, where you split real text with a real tokenizer.