# Top-K

Course: [Large Language Models](https://www.devtutorials.ai/learn/llm) · Section: Sampling
Source: https://www.devtutorials.ai/learn/llm/sampling/top-k

> How top-k sampling keeps only the k most likely tokens, cuts off the long tail of unlikely ones, and how it works together with temperature.

Temperature changes the shape of the chances, but every token still keeps a small chance. With about 150,000 tokens in the vocabulary, all those tiny chances add up. From time to time, sampling picks one of them, and the reply suddenly contains an odd word.

**Top-k** is the simplest fix: only the **k** most likely tokens are allowed. Everything else is cut out.

**The main idea:** top-k keeps the k best tokens, throws the rest away, and shares out their chance among the ones that are left.

## A Real-World Example

Think about a talent show with a hundred singers.

The judges do not let all hundred into the final. They keep the top 5. Only those 5 can win. A singer in 6th place cannot win, even with a lucky night.

Inside the final, the public still votes. The best singer out of the 5 is likely to win, but the others still have a real chance.

Top-k works in the same way.

- **The hundred singers** are all the tokens in the vocabulary.
- **Keeping the top 5** is top-k with k = 5.
- **The public vote** is sampling among the tokens that are left.
- **6th place never winning** is the long tail being cut off.

**In simple words:** top-k says "only the best k tokens may be picked".

## What You Will Learn

In this lesson, you will learn:

1. What the long tail is, and why it causes odd words.
2. How top-k keeps the k most likely tokens.
3. How the kept chances are shared out again.
4. Why k = 1 is the same as greedy decoding.
5. How top-k works together with temperature.

In the AI sandbox, the **Sampling** tab applies top-k to the real model's scores. Top-k is not one of WebLLM's settings, so the sandbox applies it in your browser, using the same steps as the code on this page. Many other AI libraries do offer a `top_k` setting.

## The Long Tail

Here is a forecast for the text "My favourite colour is". These are example chances.

- ` blue` 32%
- ` green` 20%
- ` red` 15%
- ` purple` 10%
- ` black` 8%
- ` the` 6%
- ` yellow` 5%
- ` probably` 4%

The tokens at the bottom of the list are the **long tail**. Each one by itself is unlikely. But together, they are not unlikely at all.

```js runnable title="How much chance sits in the tail?"
const chances = {
  " blue": 0.32, " green": 0.20, " red": 0.15, " purple": 0.10,
  " black": 0.08, " the": 0.06, " yellow": 0.05, " probably": 0.04,
};

const sorted = Object.entries(chances).sort((a, b) => b[1] - a[1]);
const top3 = sorted.slice(0, 3).reduce((sum, [, c]) => sum + c, 0);
const tail = sorted.slice(3).reduce((sum, [, c]) => sum + c, 0);

console.log("Top 3 together:", Math.round(top3 * 100) + "%");
console.log("All the rest:  ", Math.round(tail * 100) + "%");
```

Expected output:

```text
Top 3 together: 67%
All the rest:   33%
```

About a third of the time, sampling picks a token outside the top 3. In a real model, the tail is about 150,000 tokens long, and some of those tokens make no sense in this place.

## How Top-K Works

1. Sort the tokens from most to least likely.
2. Keep the first **k** tokens. Drop all the others.
3. **Renormalize**: divide each kept chance by their total, so they add up to 100% again.

```js runnable title="Keep the top 3 and share out the chance"
const chances = {
  " blue": 0.32, " green": 0.20, " red": 0.15, " purple": 0.10,
  " black": 0.08, " the": 0.06, " yellow": 0.05, " probably": 0.04,
};

function topK(chances, k) {
  const kept = Object.entries(chances).sort((a, b) => b[1] - a[1]).slice(0, k);
  const total = kept.reduce((sum, [, c]) => sum + c, 0);
  return kept.map(([token, c]) => [token, c / total]);
}

for (const [token, chance] of topK(chances, 3)) {
  console.log(JSON.stringify(token).padEnd(8), (chance * 100).toFixed(1) + "%");
}
```

Expected output:

```text
" blue"  47.8%
" green" 29.9%
" red"   22.4%
```

The three kept tokens stay in the same order. Their chances grow, because the chance of the dropped tokens is shared out among them.

**In simple words:** cut the list at k, then make the rest add up to 100% again.

### k = 1 Is Greedy

If you keep only one token, there is nothing left to choose. The top token wins every time. So **top-k with k = 1 is greedy decoding**.

```js runnable title="Top-k with k = 1"
const chances = { " blue": 0.32, " green": 0.20, " red": 0.15 };

function topK(chances, k) {
  const kept = Object.entries(chances).sort((a, b) => b[1] - a[1]).slice(0, k);
  const total = kept.reduce((sum, [, c]) => sum + c, 0);
  return kept.map(([token, c]) => [token, c / total]);
}

console.log(topK(chances, 1).map(([t, c]) => t + " " + c * 100 + "%").join(", "));
```

Expected output:

```text
 blue 100%
```

## The Effect on Sampling

The next code box samples 1,000 times, once without top-k and once with k = 3, and counts how often a tail token was picked.

```js runnable title="How often does a tail token get picked?"
function seededRandom(seed) {
  let a = seed >>> 0;
  return () => {
    a = (a + 0x6d2b79f5) >>> 0;
    let t = a;
    t = Math.imul(t ^ (t >>> 15), t | 1);
    t ^= t + Math.imul(t ^ (t >>> 7), t | 61);
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}

const sorted = [
  [" blue", 0.32], [" green", 0.20], [" red", 0.15], [" purple", 0.10],
  [" black", 0.08], [" the", 0.06], [" yellow", 0.05], [" probably", 0.04],
];

function tailPicks(k) {
  const kept = sorted.slice(0, k);
  const total = kept.reduce((sum, [, c]) => sum + c, 0);
  const random = seededRandom(9);
  let tail = 0;
  for (let i = 0; i < 1000; i++) {
    let r = random() * total;
    let j = 0;
    while (r > kept[j][1]) { r -= kept[j][1]; j++; }
    if (j >= 3) tail++;
  }
  return tail;
}

console.log("No top-k (k = 8):", tailPicks(8), "tail picks out of 1000");
console.log("Top-k with k = 3:", tailPicks(3), "tail picks out of 1000");
```

Expected output:

```text
No top-k (k = 8): 339 tail picks out of 1000
Top-k with k = 3: 0 tail picks out of 1000
```

With top-k = 3, a tail token can never be picked.

**Easy Way to Remember:** **top-k = a fixed-size shortlist**. Only k tokens make the list. **k = 1 is greedy.**

## Top-K With Temperature

Temperature and top-k work together. The usual order is:

1. Apply **temperature** to the logits.
2. Run softmax.
3. Apply **top-k**.

A higher temperature spreads the chances out, which makes the tail bigger. Top-k then cuts that bigger tail. So you can raise the temperature for variety, and use top-k as a safety net.

```js runnable title="Temperature 1.5, then top-k 3"
const tokens = [" blue", " green", " red", " purple", " black"];
const logits = [3.0, 2.5, 2.2, 1.8, 1.6];

function chances(temperature, k) {
  const exps = logits.map((l) => Math.exp(l / temperature));
  const order = exps.map((e, i) => i).sort((a, b) => exps[b] - exps[a]).slice(0, k);
  const total = order.reduce((sum, i) => sum + exps[i], 0);
  return order.map((i) => tokens[i] + " " + Math.round(exps[i] / total * 100) + "%").join(", ");
}

console.log("T 1.5, all:  ", chances(1.5, 5));
console.log("T 1.5, k = 3:", chances(1.5, 3));
```

Expected output:

```text
T 1.5, all:    blue 32%,  green 23%,  red 19%,  purple 14%,  black 13%
T 1.5, k = 3:  blue 43%,  green 31%,  red 25%
```

## Try It in the AI Sandbox

Press the button to open the **Sampling** tab. Set **Temperature** to 1.5, then drag **Top-k** from 0 (off) down to 3, and then to 1. Watch the greyed rows grow as tokens are cut.

## The Weakness of Top-K

Top-k always keeps **exactly** k tokens, no matter what the situation is.

- When the model is **sure**, like after "The capital of France is", most of the k tokens are bad choices. Keeping 40 of them lets nonsense in.
- When the model is **unsure**, like after "My favourite food is", many good choices exist. Keeping only 5 cuts good ones out.

A fixed k cannot be right for both cases. The next lesson, **Top-P**, fixes this by changing the size of the shortlist at every step.

A small k with a high temperature can still give strange results, because all k tokens may end up with similar chances. Also, `top_k` is not available everywhere. Check which settings your AI service or library supports before you rely on it.

## Common mistakes

| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| Forgetting to renormalize | "Just delete the tail tokens." | The kept chances must be divided by their total. | The chances add up to less than 100%, and sampling breaks. |
| A tiny k for stories | "k = 2 keeps the text safe." | Only two choices per step makes text dull and repetitive. | Stories feel flat and loop. |
| A huge k | "k = 1,000 is safer." | Such a big k cuts almost nothing. | Odd tail tokens still appear. |
| Thinking k adapts | "Top-k keeps fewer tokens when the model is sure." | Top-k always keeps exactly k tokens. | Bad tokens get in when the model is sure. |
| Wrong order | "Apply top-k before temperature." | The usual order is temperature, then top-k. | Results differ from what the library does. |
| Expecting top_k everywhere | "Every API has top_k." | Some, like WebLLM, do not. | Your setting is ignored or rejected. |

## Exercises

Each exercise ends with a value on the last line. Press **Check** to compare your answer with the expected one.

## What to Remember

- The **long tail** is the many unlikely tokens. Together, they can hold a lot of chance.
- **Top-k** keeps only the **k** most likely tokens.
- The kept chances are **renormalized** so they add up to 100%.
- **k = 1** is greedy decoding.
- The usual order is **temperature, then top-k**.
- Top-k keeps the **same number** of tokens at every step, which is its weakness.
- WebLLM has no `top_k` setting. The sandbox applies it in the browser.

Next: **Top-P**, a shortlist that grows and shrinks based on how sure the model is.