Token Explorer

By the Syncfusion DevTutorials team · Last updated October 20, 2026

Use a real LLM tokenizer in your browser to see how text, spaces, numbers, code and other languages split into tokens, and work with token counts.

So far, you have used small, made-up token lists. Now it is time for the real thing.

The lab on this page loads the real tokenizer of the model you will run later in this course, Qwen2.5. Everything it shows is worked out live, in your browser, from the text you type.

The main idea: the best way to understand tokens is to type text and watch how a real tokenizer splits it.

A Real-World Example

Think about a kitchen scale. You can read that a cup of rice weighs "about 200 grams". But you only really learn it when you put different things on the scale yourself: rice, sugar, flour, a full cup and a half cup.

The Token Explorer is a scale for text.

• Putting food on the scale is typing text into the lab.
• The number on the display is the token count.
• Trying different foods is trying sentences, code, numbers, and other languages.

In simple words: you learn how "heavy" text is in tokens by weighing it yourself.

What You Will Learn

In this lesson, you will learn:

1. How to use a real tokenizer in your browser.
2. How spaces and capital letters change tokens.
3. How numbers, code, and other languages split.
4. How emoji can split into parts of a character.
5. How to read the token format used inside the tokenizer.
6. How to work out characters per token and token budgets.

The Token Explorer

Press Load the tokenizer, then type in the box. Each coloured chip is one token. Tick Show token IDs to see the number under each token.

How to read the lab:

• Each chip is one token. The colours only help you see where one token ends and the next begins.
• A dot (·) is a space. A ↵ is a new line.
• A dashed chip like ‹f0› holds raw bytes that are only part of a character.
• A dark chip is a special token.
• The numbers above the chips are the token count, the character count, the word count, and characters per token.

Things to Try

Use the example buttons, or type your own text. For each one, look at the chips and the token count.

Spaces

Type cat and then cat with a space in front. Look at the IDs.

• The space is joined to the front of the next word.
• cat and cat are different tokens with different IDs.
• Two spaces in a row may become their own token.

Capital Letters

Press Spaces & capitals. Compare cat, Cat, and CAT.

• Each form can have a different ID.
• Common forms are often one token. Rare forms, like all capitals, may split into several.

Numbers

Press Numbers. Look at how long numbers split.

• A long number is not one token.
• It splits into small groups of digits, or even single digits.
• This is one reason LLMs make mistakes with big sums.

Code

Press Code. Look at the spaces at the start of the second line.

• Brackets and symbols are tokens too.
• The spaces used to indent code also cost tokens.

Other Languages

Press Hindi, then compare its token count with the English sentence.

• Many languages need more tokens for the same meaning.
• That means they fill the context window faster and cost more on paid services.

Emoji and Rare Characters

Press Emoji. Some emoji may show as dashed chips.

• An emoji is stored as several bytes.
• If the tokenizer has no single token for it, the emoji is split into byte pieces.
• Each piece alone is not a full character, so the lab shows its bytes, like ‹f0›.

How Tokens Look Inside

Inside the tokenizer, each token is stored as a string. A space at the start of a token is stored as a special character, Ġ. A new line is stored as Ċ.

The next boxes use token lists written in this inside format.

Count the Tokens That Start With a Space

Which tokens start with a space?

const tokens = ["The", "Ġweather", "Ġis", "Ġlovely", "Ġtoday", "."];

let withSpace = 0;
for (const token of tokens) {
  if (token.startsWith("Ġ")) withSpace++;
}

console.log("Tokens:", tokens.length);
console.log("Start with a space:", withSpace);

Expected output:

Tokens: 6
Start with a space: 4

The first word has no space in front, because nothing comes before it. Every later word carries its own space.

Turn the Tokens Back Into Text

To decode this format, swap Ġ for a space, swap Ċ for a new line, and join the tokens with nothing between them.

Decode the inside format

const tokens = ["Hello", "Ġthere", "!", "Ċ", "How", "Ġare", "Ġyou", "?"];

const text = tokens
  .map((token) => token.replaceAll("Ġ", " ").replaceAll("Ċ", "\n"))
  .join("");

console.log(text);

Expected output:

Hello there!
How are you?

Real tokenizers do the same, plus one more step for bytes that are not plain English letters.

Working With Token Counts

Once you can count tokens, you can plan with them. The counts in the next boxes are example numbers. Replace them with the real counts the lab shows for your own text.

Characters per Token

Characters per token tells you how much text fits in one token. A higher number means the text is cheaper to send.

Characters per token

const samples = [
  { name: "English sentence", characters: 28, tokens: 6 },
  { name: "Code", characters: 38, tokens: 14 },
  { name: "Other language", characters: 22, tokens: 15 },
];

for (const sample of samples) {
  const perToken = (sample.characters / sample.tokens).toFixed(1);
  console.log(sample.name + ":", perToken, "characters per token");
}

Expected output:

English sentence: 4.7 characters per token
Code: 2.7 characters per token
Other language: 1.5 characters per token

With these example numbers, English packs the most text into each token. Try the same three kinds of text in the lab and put your real counts in the box.

Will It Fit?

Say you want to send a document with several sections. Each number is the token count of one section. How many sections fit before you use up 3,000 tokens?

How many sections fit in the budget?

const sectionTokens = [800, 650, 900, 700, 500];
const budget = 3000;

let used = 0;
let fitted = 0;
for (const tokens of sectionTokens) {
  if (used + tokens > budget) break;
  used += tokens;
  fitted++;
}

console.log("Sections that fit:", fitted);
console.log("Tokens used:", used);

Expected output:

Sections that fit: 3
Tokens used: 2350

The fourth section would bring the total to 3,050, which is over the budget. So only three fit.

Comparing Languages

The same short message, sent many times, in two languages. The token counts are examples.

The same message in two languages

const englishTokens = 12;
const otherTokens = 30;
const messages = 1000;

console.log("English total:", englishTokens * messages, "tokens");
console.log("Other language total:", otherTokens * messages, "tokens");
console.log("Times more:", otherTokens / englishTokens);

Expected output:

English total: 12000 tokens
Other language total: 30000 tokens
Times more: 2.5

In this example, the same message costs 2.5 times as many tokens. Check your own language in the lab. The real number depends on the tokenizer.

Common mistakes

Exercises

Each exercise ends with a value on the last line. Press Check to compare it with the expected answer.

What to Remember

• The lab uses the real Qwen2.5 tokenizer, live in your browser.
• Spaces stick to the next word, so cat and cat are different tokens.
• Capital letters change tokens too.
• Long numbers split into small groups or single digits.
• Code and other languages often need more tokens.
• Emoji can split into byte tokens.
• Inside the tokenizer, Ġ means a space and Ċ means a new line.
• Token counts belong to one tokenizer. Count with the model you use.

Next: Embeddings, how a model turns each token ID into numbers that carry meaning.