Tokens in Large Language Models

Lessons in this section

1. Tokens

   What a token is, why LLMs read tokens instead of letters or words, token IDs and the vocabulary, and why token counts matter.

2. Tokenization

   How a tokenizer learns its pieces with byte pair encoding (BPE), how it splits new words, and how special tokens mark the chat format.

3. Token Explorer

   Use a real LLM tokenizer in your browser to see how text, spaces, numbers, code and other languages split into tokens, and work with token counts.