# Token Explorer

Course: [Large Language Models](https://www.devtutorials.ai/learn/llm) · Section: Tokens
Source: https://www.devtutorials.ai/learn/llm/tokens/token-explorer

> Use a real LLM tokenizer in your browser to see how text, spaces, numbers, code and other languages split into tokens, and work with token counts.

So far, you have used small, made-up token lists. Now it is time for the real thing.

The lab on this page loads the **real tokenizer** of the model you will run later in this course, Qwen2.5. Everything it shows is worked out live, in your browser, from the text you type.

**The main idea:** the best way to understand tokens is to type text and watch how a real tokenizer splits it.

## A Real-World Example

Think about a kitchen scale. You can read that a cup of rice weighs "about 200 grams". But you only really learn it when you put different things on the scale yourself: rice, sugar, flour, a full cup and a half cup.

The Token Explorer is a scale for text.

- **Putting food on the scale** is typing text into the lab.
- **The number on the display** is the token count.
- **Trying different foods** is trying sentences, code, numbers, and other languages.

**In simple words:** you learn how "heavy" text is in tokens by weighing it yourself.

## What You Will Learn

In this lesson, you will learn:

1. How to use a real tokenizer in your browser.
2. How spaces and capital letters change tokens.
3. How numbers, code, and other languages split.
4. How emoji can split into parts of a character.
5. How to read the token format used inside the tokenizer.
6. How to work out characters per token and token budgets.

The lab downloads the tokenizer files once (about 7 MB). If your connection blocks the download, the lab shows an error and no tokens. It never shows made-up results.

## The Token Explorer

Press **Load the tokenizer**, then type in the box. Each coloured chip is one token. Tick **Show token IDs** to see the number under each token.

How to read the lab:

- **Each chip** is one token. The colours only help you see where one token ends and the next begins.
- **A dot (·)** is a space. A **↵** is a new line.
- **A dashed chip** like ‹f0› holds raw bytes that are only part of a character.
- **A dark chip** is a special token.
- **The numbers** above the chips are the token count, the character count, the word count, and characters per token.

## Things to Try

Use the example buttons, or type your own text. For each one, look at the chips and the token count.

### Spaces

Type `cat` and then ` cat` with a space in front. Look at the IDs.

- The space is joined to the **front** of the next word.
- `cat` and ` cat` are **different tokens** with different IDs.
- Two spaces in a row may become their own token.

### Capital Letters

Press **Spaces & capitals**. Compare `cat`, `Cat`, and `CAT`.

- Each form can have a **different ID**.
- Common forms are often one token. Rare forms, like all capitals, may split into several.

### Numbers

Press **Numbers**. Look at how long numbers split.

- A long number is **not** one token.
- It splits into small groups of digits, or even single digits.
- This is one reason LLMs make mistakes with big sums.

### Code

Press **Code**. Look at the spaces at the start of the second line.

- Brackets and symbols are tokens too.
- The spaces used to indent code also cost tokens.

### Other Languages

Press **Hindi**, then compare its token count with the English sentence.

- Many languages need **more tokens** for the same meaning.
- That means they fill the context window faster and cost more on paid services.

### Emoji and Rare Characters

Press **Emoji**. Some emoji may show as dashed chips.

- An emoji is stored as several **bytes**.
- If the tokenizer has no single token for it, the emoji is split into byte pieces.
- Each piece alone is not a full character, so the lab shows its bytes, like ‹f0›.

**Easy Way to Remember:** **spaces stick to the next word**, **numbers break into bits**, and **other languages cost more**.

## How Tokens Look Inside

Inside the tokenizer, each token is stored as a string. A space at the start of a token is stored as a special character, `Ġ`. A new line is stored as `Ċ`.

The next boxes use token lists written in this inside format.

### Count the Tokens That Start With a Space

```js runnable title="Which tokens start with a space?"
const tokens = ["The", "Ġweather", "Ġis", "Ġlovely", "Ġtoday", "."];

let withSpace = 0;
for (const token of tokens) {
  if (token.startsWith("Ġ")) withSpace++;
}

console.log("Tokens:", tokens.length);
console.log("Start with a space:", withSpace);
```

Expected output:

```text
Tokens: 6
Start with a space: 4
```

The first word has no space in front, because nothing comes before it. Every later word carries its own space.

### Turn the Tokens Back Into Text

To decode this format, swap `Ġ` for a space, swap `Ċ` for a new line, and join the tokens with nothing between them.

```js runnable title="Decode the inside format"
const tokens = ["Hello", "Ġthere", "!", "Ċ", "How", "Ġare", "Ġyou", "?"];

const text = tokens
  .map((token) => token.replaceAll("Ġ", " ").replaceAll("Ċ", "\n"))
  .join("");

console.log(text);
```

Expected output:

```text
Hello there!
How are you?
```

Real tokenizers do the same, plus one more step for bytes that are not plain English letters.

## Working With Token Counts

Once you can count tokens, you can plan with them. The counts in the next boxes are **example numbers**. Replace them with the real counts the lab shows for your own text.

### Characters per Token

**Characters per token** tells you how much text fits in one token. A higher number means the text is cheaper to send.

```js runnable title="Characters per token"
const samples = [
  { name: "English sentence", characters: 28, tokens: 6 },
  { name: "Code", characters: 38, tokens: 14 },
  { name: "Other language", characters: 22, tokens: 15 },
];

for (const sample of samples) {
  const perToken = (sample.characters / sample.tokens).toFixed(1);
  console.log(sample.name + ":", perToken, "characters per token");
}
```

Expected output:

```text
English sentence: 4.7 characters per token
Code: 2.7 characters per token
Other language: 1.5 characters per token
```

With these example numbers, English packs the most text into each token. Try the same three kinds of text in the lab and put your real counts in the box.

### Will It Fit?

Say you want to send a document with several sections. Each number is the token count of one section. How many sections fit before you use up 3,000 tokens?

```js runnable title="How many sections fit in the budget?"
const sectionTokens = [800, 650, 900, 700, 500];
const budget = 3000;

let used = 0;
let fitted = 0;
for (const tokens of sectionTokens) {
  if (used + tokens > budget) break;
  used += tokens;
  fitted++;
}

console.log("Sections that fit:", fitted);
console.log("Tokens used:", used);
```

Expected output:

```text
Sections that fit: 3
Tokens used: 2350
```

The fourth section would bring the total to 3,050, which is over the budget. So only three fit.

### Comparing Languages

The same short message, sent many times, in two languages. The token counts are examples.

```js runnable title="The same message in two languages"
const englishTokens = 12;
const otherTokens = 30;
const messages = 1000;

console.log("English total:", englishTokens * messages, "tokens");
console.log("Other language total:", otherTokens * messages, "tokens");
console.log("Times more:", otherTokens / englishTokens);
```

Expected output:

```text
English total: 12000 tokens
Other language total: 30000 tokens
Times more: 2.5
```

In this example, the same message costs 2.5 times as many tokens. Check your own language in the lab. The real number depends on the tokenizer.

Token counts only apply to **one** tokenizer. The counts in this lab are for Qwen2.5. Another model, like a GPT or Llama model, may split the same text differently. When cost or limits matter, count with the tokenizer of the model you actually use.

## Common mistakes

| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| Ignoring the leading space | "`Hello` and ` Hello` are the same token." | The space is part of the token, so the IDs differ. | Token-level checks and limits fail in odd ways. |
| Treating numbers as one token | "`1234567` is one token." | Long numbers split into small groups or single digits. | Maths and number tasks go wrong, and numbers cost more than expected. |
| Assuming all languages cost the same | "A message costs the same in any language." | Many languages need more tokens for the same meaning. | Apps in those languages are slower and more expensive. |
| Forgetting code indentation | "Spaces in code are free." | Indentation and symbols are tokens too. | Large code files fill the context window quickly. |
| Reading byte tokens as broken text | "The dashed chips mean the tokenizer failed." | An emoji or rare character can split into byte tokens. | You think the text is corrupt when it is fine. |
| Reusing counts across models | "My count from this lab works for every model." | Each model has its own tokenizer and its own counts. | Budgets and cost estimates are wrong for the other model. |

## Exercises

Each exercise ends with a value on the last line. Press **Check** to compare it with the expected answer.

## What to Remember

- The lab uses the **real Qwen2.5 tokenizer**, live in your browser.
- **Spaces stick to the next word**, so ` cat` and `cat` are different tokens.
- **Capital letters** change tokens too.
- **Long numbers** split into small groups or single digits.
- **Code** and **other languages** often need more tokens.
- **Emoji** can split into byte tokens.
- Inside the tokenizer, **Ġ** means a space and **Ċ** means a new line.
- Token counts belong to **one tokenizer**. Count with the model you use.

Next: **Embeddings**, how a model turns each token ID into numbers that carry meaning.