# Generation Flow

Course: [Large Language Models](https://www.devtutorials.ai/learn/llm) · Section: LLM Essentials
Source: https://www.devtutorials.ai/learn/llm/llm-essentials/generation-flow

> The full path from the prompt you type to the reply you read: chat messages, tokens, the model, picking a token, the loop, stopping, and streaming.

You type a question and press Enter. A short time later, the answer starts to show up, one small part at a time.

A lot is happening in that moment. This lesson will walk with your question through every step, from when you send it to the very last word of the reply.

**The main idea:** a reply is built in a loop. The model reads all the text, picks the next piece, adds it, and then reads everything again.

## A Real-World Example

Imagine a friend who helps you finish a story, one word at a time.

You read the story out loud. Your friend hears the whole thing and says one new word. You write that word down. Then you read the whole story out loud again, with the new word added at the end. Your friend says the next word.

You keep doing this until your friend says "The end", or until the page is full.

An LLM works just like that friend.

- The **story so far** is your question plus the reply written so far.
- Your **friend saying one word** is the model picking the next piece.
- **Writing it down** is adding that piece to the text.
- **"The end"** or **a full page** are the two ways the reply can stop.

**In simple words:** the model writes a reply by repeating "read everything, add one piece" again and again until it is told to stop.

## What You Will Learn

In this lesson, you will learn:

1. The six steps between your question and the reply.
2. How chat messages are joined into one piece of text.
3. Why the text is split into small pieces called tokens.
4. How the generation loop works, and the two ways it stops.
5. Why replies show up word by word.
6. Why the whole chat is sent again every time.

The code boxes on this page use plain JavaScript and a tiny pretend model. They show each step on its own so it is easy to follow. From the Predictions chapter on, you will run the same steps with a real model in your browser.

## The Six Steps at a Glance

Here is the full path of one reply. Each step has its own section below.

1. **Messages.** Your question, plus any instructions, are put together as a list of messages.
2. **Chat format.** The messages are joined into one long piece of text.
3. **Tokens.** That text is split into small pieces called tokens.
4. **Model.** The model reads all the tokens and gives a chance to every possible next token.
5. **Pick and add.** One token is picked and added to the end. Then steps 4 and 5 run again.
6. **Text out.** When the loop stops, the tokens are turned back into text that you can read.

## Step 1: Messages

A chat is more than just your question. It is a list of **messages**. Each message has a **role** that tells who wrote it.

- **system**: instructions from the app, like "You are a helpful support assistant. Keep answers short." You usually do not see this message.
- **user**: what you type.
- **assistant**: what the model replied earlier in the chat.

```js runnable title="A chat is a list of messages"
const messages = [
  { role: "system", content: "You are a helpful assistant. Keep answers short." },
  { role: "user", content: "What is an LLM?" },
];

for (const message of messages) {
  console.log(message.role + ":", message.content);
}
```

Expected output:

```text
system: You are a helpful assistant. Keep answers short.
user: What is an LLM?
```

The system message sets the rules for the whole chat. The user message is the question.

**In simple words:** before anything else, your question is put into a list, right next to the app's instructions.

## Step 2: Chat Format

The model cannot read a list. It can only read one long piece of text. So the messages are joined together, with special markers that show where each message begins and ends.

This joining is called the **chat format**, or **chat template**. Each model family uses its own markers. The idea is always the same.

```js runnable title="Join the messages into one piece of text"
const messages = [
  { role: "system", content: "Keep answers short." },
  { role: "user", content: "What is an LLM?" },
];

let text = "";
for (const message of messages) {
  text += "<|" + message.role + "|>" + message.content + "<|end|>\n";
}
text += "<|assistant|>";

console.log(text);
```

Expected output:

```text
<|system|>Keep answers short.<|end|>
<|user|>What is an LLM?<|end|>
<|assistant|>
```

Look at the last line. The text ends with the assistant marker and nothing after it. That is an empty space. The model's job is to fill it in.

**In simple words:** the chat becomes a script with a blank line at the end for the assistant to "speak".

## Step 3: Tokens

The model does not read letters or whole words. It reads **tokens**. A token is a small piece of text, often a whole word or a part of a word. Every token has a number, called its **token ID**.

The next box uses a tiny made-up list of tokens. It splits a sentence into words and looks up each word's ID.

```js runnable title="Turn text into token IDs"
const vocabulary = { "what": 1, "is": 2, "an": 3, "llm": 4, "?": 5 };

const text = "what is an llm ?";
const pieces = text.split(" ");
const ids = pieces.map((piece) => vocabulary[piece]);

console.log("Pieces:", pieces.join(" | "));
console.log("IDs:   ", ids.join(", "));
```

Expected output:

```text
Pieces: what | is | an | llm | ?
IDs:    1, 2, 3, 4, 5
```

Real tokenizers are smarter than this. They split rare words into smaller parts, and a real list has tens of thousands of tokens. The Tokens chapter shows a real one.

**In simple words:** the model sees your text as a list of numbers.

## Steps 4 and 5: The Generation Loop

Now the model gets to work. This is the loop from the LLM Basics lesson.

1. The model reads **all** the tokens so far.
2. It gives every possible next token a chance.
3. One token is picked.
4. That token is added to the end.
5. Go back to step 1.

Each pass through the loop adds **exactly one token**. A reply of 200 tokens needs 200 passes.

### The Two Ways It Stops

The loop cannot run forever. It stops in one of two ways.

- **The end token.** The model has a special token that means "I am done". When the model picks it, the reply is finished. This is the normal way to stop.
- **The maximum length.** The app sets a limit on how many tokens the reply may have. This is often called **max tokens**. If the limit is reached, the reply stops, even in the middle of a sentence.

The next box has a pretend model. It always gives the same next word. The loop stops at the end token.

```js runnable title="Loop until the end token"
const nextWord = {
  "<|assistant|>": "An",
  "An": "LLM",
  "LLM": "predicts",
  "predicts": "text.",
  "text.": "<end>",
};

let last = "<|assistant|>";
const reply = [];

for (let step = 1; step <= 10; step++) {
  const word = nextWord[last];
  if (word === "<end>") {
    console.log("Step " + step + ": end token, stop");
    break;
  }
  reply.push(word);
  console.log("Step " + step + ": added", word);
  last = word;
}

console.log("Reply:", reply.join(" "));
```

Expected output:

```text
Step 1: added An
Step 2: added LLM
Step 3: added predicts
Step 4: added text.
Step 5: end token, stop
Reply: An LLM predicts text.
```

The loop could have run 10 times. It stopped after step 5 because the model picked the end token.

Now look at the other way to stop. Here, max tokens is set to 2.

```js runnable title="Stop at max tokens"
const nextWord = {
  "<|assistant|>": "An",
  "An": "LLM",
  "LLM": "predicts",
  "predicts": "text.",
  "text.": "<end>",
};

const maxTokens = 2;
let last = "<|assistant|>";
const reply = [];

while (reply.length < maxTokens) {
  const word = nextWord[last];
  if (word === "<end>") break;
  reply.push(word);
  last = word;
}

console.log("Reply:", reply.join(" "));
console.log("Stopped because max tokens is", maxTokens);
```

Expected output:

```text
Reply: An LLM
Stopped because max tokens is 2
```

The reply was cut off in the middle. If you ever see a reply that ends halfway through a sentence, the max tokens limit is the most likely reason.

**Easy Way to Remember:** the loop is **read, pick, add, repeat**. It ends at the **end token** or at **max tokens**, whichever comes first.

## Step 6: Text Out and Streaming

The tokens are numbers, so they are turned back into text. This is called **decoding**.

Apps do not wait for the whole reply. Each new token is shown as soon as it is picked. This is called **streaming**. It is why replies appear word by word, like someone typing.

Streaming does not make the model faster. It only lets you start reading sooner.

## Why the Whole Chat Is Sent Again

Here is something that surprises many people. The model does not remember your chat. Every time you send a message, the app sends the **whole chat so far** again.

```js runnable title="Each turn sends the full history"
const history = [{ role: "system", content: "Keep answers short." }];

function sendTurn(question, answer) {
  history.push({ role: "user", content: question });
  console.log("Sending", history.length, "messages to the model");
  history.push({ role: "assistant", content: answer });
}

sendTurn("What is an LLM?", "A program that predicts text.");
sendTurn("Who makes them?", "Many companies and research groups.");
sendTurn("Can I run one at home?", "Yes, small ones.");
```

Expected output:

```text
Sending 2 messages to the model
Sending 4 messages to the model
Sending 6 messages to the model
```

The list grows with every turn. The model reads all of it each time. This is why a very long chat can get slow, and why it can run out of room. The next lesson, Model Limits, explains that room.

**In simple words:** the model's "memory" is just the app sending the old messages again.

A long chat costs more with every turn. The model reads the whole history each time, so a 30-message chat makes it read all 30 messages just to answer the next one. When a new topic starts, start a new chat.

## Common mistakes

| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| Thinking the reply is written all at once | "The model plans the full answer, then shows it." | It writes one token at a time, reading everything before each one. | You expect it to fix an early part of the answer later. It cannot. |
| Ignoring the system message | "Only my question matters." | The system message sets rules the model follows for the whole chat. | Answers come out too long, in the wrong tone, or off-topic. |
| Setting max tokens too low | "A small limit keeps answers short and neat." | The limit cuts the reply off, even mid-sentence. | Replies end halfway through a word or a code block. |
| Thinking the model has memory | "It remembers what I said earlier." | The app sends the earlier messages again each time. | A new chat, or a trimmed history, makes it "forget". |
| Thinking streaming is faster | "Streaming makes the model answer quicker." | Streaming only shows each token as soon as it is ready. | You blame the network when the model itself is slow. |
| Thinking the model reads words | "It sees my sentence the way I do." | It sees token IDs, which are numbers for pieces of text. | Spelling and letter-counting questions confuse it. |

## Exercises

Each exercise ends with a value on the last line. Press **Check** to compare it with the expected answer.

## What to Remember

- A chat is a list of **messages** with roles: **system**, **user**, and **assistant**.
- The messages are joined into one text with the **chat format**.
- The text is split into **tokens**, and each token has a number called a **token ID**.
- The **generation loop** reads all tokens, picks one, adds it, and repeats.
- The loop stops at the **end token** or at **max tokens**.
- **Streaming** shows each token as soon as it is picked.
- The model has no memory. The app sends the **whole chat again** on every turn.

Next: **Model Limits**, the size of the model's working space, and the other limits every LLM has.