Sampling

The Probability Distribution article ended with a full, standardized set of probabilities across every possible next token, ready to be used. This final article in the phase covers exactly how the model actually picks one: sampling.

The simple definition

Sampling is the process of actually selecting one specific token from the probability distribution, to generate as the next piece of output. Recall from the Prediction article’s earlier mention of temperature: a model doesn’t always simply pick whichever token has the single highest probability. Sampling is the general name for the different strategies a model can use to make this selection — and the specific strategy chosen has a real, noticeable effect on how the final generated text actually reads.

Why not just always pick the highest-probability token?

This is worth answering directly, since it seems like the obvious, safest choice at first glance. Always picking the single highest-probability token — a strategy called greedy decoding — genuinely does work, and produces safe, predictable output. But it has a real, well-documented downside: text generated this way tends to feel repetitive, bland, and mechanically “safe,” since the model always takes the most statistically obvious path at every single step, never allowing for the kind of natural variation real human writing has. Real language isn’t perfectly predictable — people make word choices that are reasonable but not always the single most statistically common option, and greedy decoding, by never deviating from the top choice, tends to produce text that reads as noticeably stiffer and more repetitive than natural writing.

flowchart LR
    A[Probability Distribution] --> B{Sampling strategy}
    B -->|Greedy: always pick #1| C[Safe, but repetitive and mechanical]
    B -->|Sampling with randomness| D[More natural, varied, human-like]

How randomized sampling actually works

Rather than always picking the top choice, sampling introduces controlled randomness — treating the probability distribution literally as a set of odds, and randomly drawing one token according to those odds, the same way a weighted lottery works. A token with 60% probability gets picked about 60% of the time; a token with 5% probability gets picked only about 5% of the time, but it genuinely can still get picked occasionally, introducing natural, human-like variation into the output.

Temperature: the setting that controls how much randomness

Recall from the Prediction article’s introduction of temperature: this setting directly controls how “adventurous” the sampling process is. A low temperature sharpens the probability distribution further, making the model even more likely to pick its top choice — closer to greedy decoding, safer and more predictable. A high temperature flattens the distribution, giving lower-probability tokens a more meaningful chance of being picked — more varied, creative, and occasionally stranger output. This is a real, adjustable setting available in most AI developer tools, exactly as the Prediction article established.

flowchart LR
    A[Low temperature] --> B[Sharper distribution, safer, more repetitive]
    C[High temperature] --> D[Flatter distribution, more varied, occasionally odd]

Other real sampling techniques worth knowing

Beyond plain temperature adjustment, a couple of other named techniques are worth recognizing, since they show up in real API documentation and developer tools. Top-k sampling restricts the random draw to only the k highest-probability tokens (say, the top 40), completely ignoring the long tail of extremely unlikely options — preventing the model from ever picking something wildly implausible, while still allowing meaningful variety among genuinely reasonable choices. Top-p sampling (also called nucleus sampling) works similarly but dynamically adjusts how many tokens are considered based on the shape of the distribution itself — including just enough of the highest-probability tokens to reach a target cumulative probability (like 90%), which naturally includes more options when the distribution is flat and fewer when it’s sharply peaked.

A concrete example, layered

For a simple beginner example: given “My favorite season is”, greedy decoding might always produce “fall” if that happens to be the single most statistically common answer in training data, every single time, regardless of who’s asking; sampling with moderate temperature would sometimes produce “fall,” but other times “summer” or “spring,” reflecting the genuine variety of reasonable answers, the same way different real people would answer differently. For a production example: OpenAI’s API documentation exposes both a temperature parameter and a top_p parameter directly to developers, letting them choose, for their specific application, whether they want a customer-support bot’s answers to stay tightly consistent and predictable (low temperature) or want a creative-writing tool to produce more varied, surprising output (higher temperature) — a real, practical engineering decision made differently depending on the product being built.

Why sampling connects directly back to the Seed article

This is worth tying together explicitly, since it resolves something that might otherwise feel contradictory. Recall from the Seed article that setting a fixed seed makes a model’s “randomness” fully reproducible. Sampling’s randomness works the same way — it’s genuine, useful randomness for producing varied, natural-feeling output, but it’s driven by the same pseudo-random number generation described in the Seed article, meaning a fixed seed combined with a fixed temperature will reproduce the exact same sampled output every time, even though the process feels random in ordinary use.

Sample from a distribution with numbers

Given:

blue  = 66.5%
clear = 24.5%
dark  =  9.0%

Imagine a random number between 0 and 1:

Random interval	Selected token
0.000–0.664	blue
0.665–0.909	clear
0.910–0.999	dark

If the random draw is 0.72, the model selects clear even though blue had the highest probability. Sampling follows the distribution; it does not always choose the maximum.

Temperature reshapes the same logits

Using logits [2, 1, 0]:

Temperature	Approximate probabilities	Effect
0.5	[86.7%, 11.7%, 1.6%]	Sharper and more focused
1.0	[66.5%, 24.5%, 9.0%]	Original softmax distribution
2.0	[50.6%, 30.7%, 18.6%]	Flatter and more varied

Temperature divides the logits before softmax. It does not add knowledge or make the model reason more accurately.

Top-p with a concrete distribution

Suppose sorted probabilities are:

A 50%, B 25%, C 15%, D 7%, E 3%

With top_p = 0.80, A and B reach only 75%, so C is also included. Sampling occurs among A, B, and C after renormalization; D and E are excluded for this step.

Real GPT and Gemini controls

OpenAI’s generation API documentation describes temperature from 0 to 2 and top_p from 0 to 1 for endpoints that expose those controls, while recommending changing one rather than both together.

The Gemini generation API documents temperature, top-p, and—on supporting models—top-k. Defaults and supported controls vary by model, so production code should read the model’s current documentation rather than copying settings blindly.

GPT/Gemini logits → filters and temperature → sampling → next token

Common misconception

A frequent beginner assumption: that sampling randomness means a model is somehow “guessing wildly” or being unreliable when it doesn’t pick the most likely answer. As this article has explained, sampling is a deliberate, carefully controlled design choice — not the model failing to find its best answer, but a genuine trade-off, tuned via temperature and related settings, between safe predictability and natural, human-like variation, chosen deliberately by whoever is building the specific application.

Closing out this phase

This article completes the Language Models phase, and it’s worth tracing the full arc it built: a Language Model predicts language, scaled up into a Large Language Model (LLM), often built and released as a broadly adaptable Foundation Model, performing the task-category known as Generative AI. Mechanically, all of this runs on Next-Token Prediction: the model calculates raw Logits for every possible token, converts them into a full Probability Distribution via softmax, and finally uses sampling to actually select the next token to generate — repeated, one token at a time, to produce every response a modern AI system has ever written. From here, the glossary is positioned to explore the specific architecture — the Transformer and its attention mechanism — that makes all of this genuinely work at scale.

In one sentence

Sampling is how a model actually selects one token from its calculated probability distribution — trading the safe, repetitive predictability of always picking the top choice for the natural, human-like variation that controlled randomness, tuned through settings like temperature, provides instead.

Related Terms

• Probability Distribution
• Prediction
• Seed
• Next-Token Prediction
• Logits