Generation Loop

By the Syncfusion DevTutorials team · Last updated October 20, 2026

Watch a real model write text one token at a time, see the choices at every step, how the loop stops, why greedy decoding can repeat, and how streaming works.

You now know how the model makes one prediction: it gets logits, runs softmax, and gives a chance for every token. A full reply is just that one step, repeated over and over.

In this lesson, you will watch the real model in your browser write text, one token at a time. You can pause after every step and see the choices it had at that moment.

The main idea: generation is a loop. The model predicts one token, adds it to the text, and then predicts again using the longer text. It keeps going until it reaches the end token or hits the limit.

A Real-World Example

Think about a game of dominoes where each new tile must match the last one played.

You look at the end of the line and choose a tile that fits there. You place it. Now the line has a new end, so you choose again. The line grows one tile at a time, and every choice depends on the tile that came before it.

You stop when you play your last tile, or when the table runs out of room.

The generation loop works in exactly the same way.

• The line of tiles is the text so far.
• Choosing a matching tile is predicting the next token.
• Placing it is adding the token to the text.
• Your last tile is the end token.
• The table running out of room is the max tokens limit.

In simple words: each new token is chosen to fit everything that was placed before it.

What You Will Learn

In this lesson, you will learn:

1. How the real model builds a reply, one token at a time.
2. How to step through each choice the model made.
3. The two ways the loop can stop.
4. Why always picking the top token can get stuck repeating itself.
5. How streaming shows tokens as soon as they are made.
6. How to estimate how long a reply will take.

Watch the Loop

Press the button below. The sandbox will open on the Step by step tab and generate a reply. Then press Play to watch the reply grow, or use Back and Next to move one step at a time. You can change the Max tokens value in the tab and press Generate to run again.

At every step:

• The text box shows everything written so far. The newest token is highlighted.
• The bars show the top 5 choices the model had at that step.
• The highlighted bar is the token it actually picked.

Things to try:

• Set Max tokens to 5 and generate a reply. The reply will be cut off.
• Try a prompt that already ends a sentence, like The sky is blue. Does the model start a new sentence, or does it stop?
• Watch for steps where the top bar is short. Those are places where the reply could easily have gone a different way.

The Loop in Code

The sandbox runs a real model. The code boxes below run the same loop using a small table of chances, so you can see every step clearly.

One Step at a Time

Greedy loop with a small table

const nextChances = {
  "a little robot": { " named": 0.6, " who": 0.3, " was": 0.1 },
  "a little robot named": { " Bolt": 0.7, " Max": 0.3 },
  "a little robot named Bolt": { " lived": 0.5, " was": 0.4, "<end>": 0.1 },
  "a little robot named Bolt lived": { "<end>": 0.9, " in": 0.1 },
};

let text = "a little robot";
for (let step = 1; step <= 10; step++) {
  const chances = nextChances[text];
  const [best] = Object.entries(chances).sort((a, b) => b[1] - a[1]);
  if (best[0] === "<end>") {
    console.log("Step " + step + ": end token");
    break;
  }
  text += best[0];
  console.log("Step " + step + ":", JSON.stringify(best[0]), "->", text);
}

Expected output:

Step 1: " named" -> a little robot named
Step 2: " Bolt" -> a little robot named Bolt
Step 3: " lived" -> a little robot named Bolt lived
Step 4: end token

Each step looks up the whole text so far. That is the key idea: the model always reads everything, not just the last token.

The Two Ways to Stop

You met the two stopping rules in the Generation Flow lesson.

Which rule stopped the loop?

function generate(tokens, maxTokens) {
  const reply = [];
  for (const token of tokens) {
    if (token === "<end>") return { reply: reply.join(""), reason: "end token" };
    if (reply.length === maxTokens) return { reply: reply.join(""), reason: "max tokens" };
    reply.push(token);
  }
  return { reply: reply.join(""), reason: "ran out" };
}

const modelTokens = ["The", " robot", " smiled", ".", "<end>"];

console.log(generate(modelTokens, 10));
console.log(generate(modelTokens, 2));

Expected output:

{
  "reply": "The robot smiled.",
  "reason": "end token"
}
{
  "reply": "The robot",
  "reason": "max tokens"
}

With room to spare, the model ends on its own. With a limit of 2, the reply is cut off early. The sandbox will tell you which rule stopped each run.

When Greedy Gets Stuck

Always picking the top token is called greedy decoding. It is simple and gives the same result every time. But it has one weakness: it can fall into a loop and repeat itself.

Greedy decoding repeats itself

const best = { "the": " cat", " cat": " sat", " sat": " on", " on": " the", " the": " cat" };

let token = "the";
const words = [token];
for (let step = 0; step < 9; step++) {
  token = best[token];
  words.push(token);
}

console.log(words.join(""));

Expected output:

the cat sat on the cat sat on the cat

Every step picked the most likely token. Together, they go round and round in a circle. Small models fall into loops like this more often, especially in long replies. You may see this happen in the sandbox when max tokens is set high.

The Sampling chapter shows how adding a little randomness when picking a token helps the model escape these loops.

Streaming

The model makes one token per step. So an app can show each token as soon as it is ready, instead of waiting for the whole reply to finish. This is called streaming.

Show the reply as it streams

const tokens = ["Hello", "!", " How", " can", " I", " help", "?"];

let shown = "";
for (const token of tokens) {
  shown += token;
  console.log(shown);
}

Expected output:

Hello
Hello!
Hello! How
Hello! How can
Hello! How can I
Hello! How can I help
Hello! How can I help?

Each line shows what the reader sees after one more token is added. In real code, an AI service sends these pieces one by one, often called chunks. You will use streaming in the LLM APIs lesson.

How Long Will It Take?

Each step takes about the same time, so you can estimate how long a full reply will take.

Estimate the time for a reply

const tokensPerSecond = 12;
const firstTokenSeconds = 0.5;

for (const replyTokens of [20, 100, 400]) {
  const seconds = firstTokenSeconds + replyTokens / tokensPerSecond;
  console.log(replyTokens, "tokens -> about", seconds.toFixed(1), "seconds");
}

Expected output:

20 tokens -> about 2.2 seconds
100 tokens -> about 8.8 seconds
400 tokens -> about 33.8 seconds

The first token takes a little extra time, because the model must first read the whole prompt. After that, each token takes about the same time. A speed of 12 tokens per second is a fair guess for a small model running in a laptop browser. Your actual speed may be different.

Common mistakes

Exercises

Each exercise ends with a value on the last line. Press Check to compare your answer with the expected one.

What to Remember

• Generation is a loop: predict one token, add it to the text, then predict again.
• Every step reads the whole text so far.
• The loop stops when it reaches the end token or when max tokens is reached.
• Greedy decoding always picks the top token. It can repeat itself.
• Streaming shows each token as soon as it is made.
• Reply time is about first-token delay + tokens ÷ tokens per second.

Next: Determinism, where you learn how to make the model give the exact same reply every time.