Sampling in Large Language Models

Lessons in this section

1. Determinism

   Why an LLM can give a different answer each time, how greedy decoding and a fixed seed make it repeat, and when repeatable output matters.

2. Temperature

   How temperature reshapes the model's chances before a token is picked: low temperature is sharp and safe, high temperature is flat and creative.

3. Top-K

   How top-k sampling keeps only the k most likely tokens, cuts off the long tail of unlikely ones, and how it works together with temperature.

4. Top-P

   How top-p (nucleus) sampling keeps the smallest set of tokens whose chances add up to p, why it adapts to how sure the model is, and how all the settings work together.