# Generation Loop

Course: [Large Language Models](https://www.devtutorials.ai/learn/llm) · Section: Predictions
Source: https://www.devtutorials.ai/learn/llm/predictions/generation-loop

> Watch a real model write text one token at a time, see the choices at every step, how the loop stops, why greedy decoding can repeat, and how streaming works.

You now know how the model makes **one** prediction: it gets logits, runs softmax, and gives a chance for every token. A full reply is just that one step, repeated over and over.

In this lesson, you will watch the real model in your browser write text, one token at a time. You can pause after every step and see the choices it had at that moment.

**The main idea:** generation is a loop. The model predicts one token, adds it to the text, and then predicts again using the longer text. It keeps going until it reaches the end token or hits the limit.

## A Real-World Example

Think about a game of dominoes where each new tile must match the last one played.

You look at the end of the line and choose a tile that fits there. You place it. Now the line has a new end, so you choose again. The line grows one tile at a time, and every choice depends on the tile that came before it.

You stop when you play your last tile, or when the table runs out of room.

The generation loop works in exactly the same way.

- **The line of tiles** is the text so far.
- **Choosing a matching tile** is predicting the next token.
- **Placing it** is adding the token to the text.
- **Your last tile** is the end token.
- **The table running out of room** is the max tokens limit.

**In simple words:** each new token is chosen to fit everything that was placed before it.

## What You Will Learn

In this lesson, you will learn:

1. How the real model builds a reply, one token at a time.
2. How to step through each choice the model made.
3. The two ways the loop can stop.
4. Why always picking the top token can get stuck repeating itself.
5. How streaming shows tokens as soon as they are made.
6. How to estimate how long a reply will take.

The AI sandbox runs the real model and records its top 5 choices at every step on its **Step by step** tab. After that, you can replay the reply one token at a time. It needs WebGPU to work. The code boxes on this page will work in any browser.

## Watch the Loop

Press the button below. The sandbox will open on the **Step by step** tab and generate a reply. Then press **Play** to watch the reply grow, or use **Back** and **Next** to move one step at a time. You can change the **Max tokens** value in the tab and press **Generate** to run again.

At every step:

- The text box shows everything written so far. The newest token is highlighted.
- The bars show the top 5 choices the model had at that step.
- The highlighted bar is the token it actually picked.

Things to try:

- Set **Max tokens** to 5 and generate a reply. The reply will be cut off.
- Try a prompt that already ends a sentence, like `The sky is blue.` Does the model start a new sentence, or does it stop?
- Watch for steps where the top bar is short. Those are places where the reply could easily have gone a different way.

## The Loop in Code

The sandbox runs a real model. The code boxes below run the same loop using a small table of chances, so you can see every step clearly.

### One Step at a Time

```js runnable title="Greedy loop with a small table"
const nextChances = {
  "a little robot": { " named": 0.6, " who": 0.3, " was": 0.1 },
  "a little robot named": { " Bolt": 0.7, " Max": 0.3 },
  "a little robot named Bolt": { " lived": 0.5, " was": 0.4, "<end>": 0.1 },
  "a little robot named Bolt lived": { "<end>": 0.9, " in": 0.1 },
};

let text = "a little robot";
for (let step = 1; step <= 10; step++) {
  const chances = nextChances[text];
  const [best] = Object.entries(chances).sort((a, b) => b[1] - a[1]);
  if (best[0] === "<end>") {
    console.log("Step " + step + ": end token");
    break;
  }
  text += best[0];
  console.log("Step " + step + ":", JSON.stringify(best[0]), "->", text);
}
```

Expected output:

```text
Step 1: " named" -> a little robot named
Step 2: " Bolt" -> a little robot named Bolt
Step 3: " lived" -> a little robot named Bolt lived
Step 4: end token
```

Each step looks up the **whole** text so far. That is the key idea: the model always reads everything, not just the last token.

### The Two Ways to Stop

You met the two stopping rules in the Generation Flow lesson.

```js runnable title="Which rule stopped the loop?"
function generate(tokens, maxTokens) {
  const reply = [];
  for (const token of tokens) {
    if (token === "<end>") return { reply: reply.join(""), reason: "end token" };
    if (reply.length === maxTokens) return { reply: reply.join(""), reason: "max tokens" };
    reply.push(token);
  }
  return { reply: reply.join(""), reason: "ran out" };
}

const modelTokens = ["The", " robot", " smiled", ".", "<end>"];

console.log(generate(modelTokens, 10));
console.log(generate(modelTokens, 2));
```

Expected output:

```text
{
  "reply": "The robot smiled.",
  "reason": "end token"
}
{
  "reply": "The robot",
  "reason": "max tokens"
}
```

With room to spare, the model ends on its own. With a limit of 2, the reply is cut off early. The sandbox will tell you which rule stopped each run.

## When Greedy Gets Stuck

Always picking the top token is called **greedy decoding**. It is simple and gives the same result every time. But it has one weakness: it can fall into a loop and repeat itself.

```js runnable title="Greedy decoding repeats itself"
const best = { "the": " cat", " cat": " sat", " sat": " on", " on": " the", " the": " cat" };

let token = "the";
const words = [token];
for (let step = 0; step < 9; step++) {
  token = best[token];
  words.push(token);
}

console.log(words.join(""));
```

Expected output:

```text
the cat sat on the cat sat on the cat
```

Every step picked the most likely token. Together, they go round and round in a circle. Small models fall into loops like this more often, especially in long replies. You may see this happen in the sandbox when max tokens is set high.

The Sampling chapter shows how adding a little randomness when picking a token helps the model escape these loops.

**Easy Way to Remember:** **predict, add, repeat**, until the **end token** or **max tokens** is reached. Greedy is safe but can **go in circles**.

## Streaming

The model makes one token per step. So an app can show each token as soon as it is ready, instead of waiting for the whole reply to finish. This is called **streaming**.

```js runnable title="Show the reply as it streams"
const tokens = ["Hello", "!", " How", " can", " I", " help", "?"];

let shown = "";
for (const token of tokens) {
  shown += token;
  console.log(shown);
}
```

Expected output:

```text
Hello
Hello!
Hello! How
Hello! How can
Hello! How can I
Hello! How can I help
Hello! How can I help?
```

Each line shows what the reader sees after one more token is added. In real code, an AI service sends these pieces one by one, often called **chunks**. You will use streaming in the LLM APIs lesson.

## How Long Will It Take?

Each step takes about the same time, so you can estimate how long a full reply will take.

```js runnable title="Estimate the time for a reply"
const tokensPerSecond = 12;
const firstTokenSeconds = 0.5;

for (const replyTokens of [20, 100, 400]) {
  const seconds = firstTokenSeconds + replyTokens / tokensPerSecond;
  console.log(replyTokens, "tokens -> about", seconds.toFixed(1), "seconds");
}
```

Expected output:

```text
20 tokens -> about 2.2 seconds
100 tokens -> about 8.8 seconds
400 tokens -> about 33.8 seconds
```

The first token takes a little extra time, because the model must first read the whole prompt. After that, each token takes about the same time. A speed of 12 tokens per second is a fair guess for a small model running in a laptop browser. Your actual speed may be different.

A long max tokens limit does not make the model write more. It only sets the most it **may** write. But it does let a looping reply run for a long time. Pick a limit that fits the answer you expect.

## Common mistakes

| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| Thinking the reply is planned | "The model decides the whole answer, then types it." | Each token is chosen at its own step, reading all the text so far. | You expect it to fix earlier words. It cannot go back. |
| Blaming the model for cut-off replies | "The model stopped mid-sentence on its own." | The reply hit max tokens. The model never got to its end token. | You change the prompt when you should raise the limit. |
| Not expecting loops | "Picking the best token every time gives the best text." | Greedy decoding can repeat the same phrase forever. | Long replies fill up with the same words. |
| Very high max tokens | "A big limit is always safer." | A looping reply runs until the limit, wasting time and tokens. | Slow, expensive replies full of repeats. |
| Streaming means faster | "Streaming makes the model answer quicker." | Streaming only shows tokens as soon as they are ready. | You look for speed in the wrong place. |
| Forgetting the first-token delay | "Every token takes the same time." | The model reads the whole prompt before the first token. | Long prompts feel slow to start, and you do not know why. |

## Exercises

Each exercise ends with a value on the last line. Press **Check** to compare your answer with the expected one.

## What to Remember

- Generation is a **loop**: predict one token, add it to the text, then predict again.
- Every step reads the **whole** text so far.
- The loop stops when it reaches the **end token** or when **max tokens** is reached.
- **Greedy decoding** always picks the top token. It can **repeat itself**.
- **Streaming** shows each token as soon as it is made.
- Reply time is about **first-token delay + tokens ÷ tokens per second**.

Next: **Determinism**, where you learn how to make the model give the exact same reply every time.