Start with the simple idea
Sampling is the step where a model chooses one result from several possible results using their predicted probabilities.
Simple learning path: problem → intuition → mechanism → example → limits
What you will learn
- Explain Sampling in plain language.
- Follow its mechanism step by step.
- Connect a small example to a real AI system.
- Recognize its strengths, limits, and common mistakes.
How this appears in current AI systems
Hugging Face Diffusers exposes modern image, video, and audio pipelines. OpenAI image generation and Google image models provide hosted examples of prompt-guided visual generation.
Official grounding: OpenAI documents its current text-generation API and Google documents the current Gemini model catalog. These pages verify available capabilities; exact model names and limits can change.
When this knowledge helps
Use Sampling when it matches the problem described below. Before choosing it, check the task, available data, quality target, cost, response time, privacy, and safety needs; popularity alone is not a reason to use it.
1. The question this module answers
Every model family covered in Level 2 eventually needs to convert a learned probability distribution into one actual, concrete output. This module covers exactly that step — sampling — which you’ve already touched on in your LLM course, now made fully explicit and connected to every modality.
2. The Problem
A model (autoregressive, diffusion — any generative model) produces a probability distribution: a set of possibilities, each with some likelihood. But you need one actual, specific output at the end. How do you go from “a distribution over possibilities” to “one concrete result”?
3. Intuition — What Does “Sampling” Actually Mean?
Imagine a weighted die — not fair, but weighted so some numbers come up more often than others. “Sampling” means rolling that die: you get one concrete outcome, but which outcome you get is influenced by the weights, not fixed in advance.
If the model predicts:
A = 0.60
B = 0.25
C = 0.10
D = 0.05
Sampling means: pick ONE of these options, where A is most likely to
be chosen (60% of the time, on average, across many samples), but B,
C, or D could really be chosen too.This is fundamentally different from just always picking the highest probability option — sampling introduces genuine variability, weighted by the learned probabilities.
4. Greedy Decoding — Always Picking the Top Option
Greedy decoding: ALWAYS select the single highest-probability option,
every time, with no randomness at all.
Given: A=0.60, B=0.25, C=0.10, D=0.05
Greedy decoding ALWAYS picks: ABehavior: completely deterministic -- the SAME input always
produces the EXACT same output, every single time
Trade-off: can feel repetitive or "safe" -- and for
autoregressive sequence generation specifically, can
sometimes lead to oddly repetitive or generic text,
since it never explores any of the other plausible
options5. Random Sampling — Really Rolling the Weighted Die
Random sampling: select an option with probability EQUAL to its
predicted probability -- really random, weighted
by the distribution.
Given: A=0.60, B=0.25, C=0.10, D=0.05
Over MANY samples: roughly 60% of the time you'd get A, roughly 25%
of the time B, roughly 10% C, roughly 5% DBehavior: really varied output across repeated generations
from the same input
Trade-off: can occasionally select a low-probability,
potentially poor-quality option (like D at only
5%) -- pure random sampling has no built-in
safeguard against occasionally choosing something
unlikely and low-quality6. Temperature — Controlling How “Confident” or “Adventurous” Sampling Is
Temperature is a single number that reshapes the probability distribution before sampling from it — without changing which option is most likely, only how sharply concentrated the distribution is around that top choice.
Low temperature (e.g., 0.2): SHARPENS the distribution --
makes the already-most-likely
option even MORE dominant,
pushing closer to greedy/
deterministic behavior
High temperature (e.g., 1.5): FLATTENS the distribution --
makes lower-probability
options relatively more
competitive, increasing
variety and "creativity"Worked example
Original distribution: A=0.60, B=0.25, C=0.10, D=0.05
At LOW temperature (0.3), reshaped roughly to:
A=0.88, B=0.10, C=0.015, D=0.005
(A now dominates even more heavily)
At HIGH temperature (1.8), reshaped roughly to:
A=0.40, B=0.28, C=0.19, D=0.13
(options are much closer to equally likely)💡 Practical intuition: low temperature = more consistent, predictable, “safe” output (good for tasks needing reliability, like structured data extraction). High temperature = more varied, exploratory, “creative” output (good for tasks like brainstorming or creative writing) — this is exactly the connection to Module 26 of the Prompt Engineering course.
Analogy: The Restaurant Menu Selection Think of setting sampling parameters like ordering food from a waiter at a restaurant:
- Greedy Decoding (Deterministic): The waiter ALWAYS brings the restaurant’s single most popular dish (e.g. Hamburger). You get the exact same plate every time you visit.
- Temperature = 0.1 (Low): The waiter is conservative. They only suggest the top 2 best-selling comfort foods. 90% of the time it’s Hamburger; 10% it’s Pizza.
- Temperature = 1.8 (High): The waiter feels chaotic. They will choose randomly from the entire kitchen, including raw cabbage, pickled eggs, or garlic dessert.
- Top-k and Top-p (The Safety Guardrails): To prevent ordering complete nonsense under high temperature, you instruct the waiter:
- Top-k: “Only show me dishes from the top 3 best-sellers.” (Limits the choices to a fixed count).
- Top-p: “Only show me dishes that represent the top 90% of total restaurant order volume.” (Limits the choices dynamically to high-quality options).
📊 Visual Flowchart: Sampling Truncation Pipeline
Here is how token probabilities are filtered before final selection:
graph TD
classDef raw fill:#3498db,stroke:#333,stroke-width:1px,color:#fff;
classDef filter fill:#f1c40f,stroke:#333,stroke-width:1px,color:#fff;
classDef selected fill:#2ecc71,stroke:#333,stroke-width:1px,color:#fff;
RawDist["Raw Predicted Probabilities:<br>['the' (60%), 'a' (25%), 'cat' (10%), 'xyz' (5%)]"]:::raw --> TempScale["1. Temperature Scale (e.g. 0.7):<br>Recalculate weights"]:::raw
TempScale --> TopKFilter{"2. Apply Top-K (e.g. K=3)"}
TopKFilter -->|Allows top 3| TopPFilter{"3. Apply Top-P (e.g. P=0.90)"}
TopKFilter -->|Discards remainder| CutLowK["Discard: 'xyz' (5%)"]
TopPFilter -->|Cumulative sum <= 90%| KeepTokens["Keep: ['the', 'a'] (sum = 85%)"]:::filter
TopPFilter -->|Exceeds P boundary| CutLowP["Discard: 'cat' (10%)"]
KeepTokens --> SampleChoice["4. Sample randomly from kept set"]:::selected
SampleChoice --> OutToken["Final Selected Token: 'a'"]:::selected7. Top-k Sampling — Limiting to the Best Few Options
Top-k sampling: only consider the TOP k highest-probability options,
ignoring everything else entirely -- then sample
(optionally with temperature) among just those k
options.
Given: A=0.60, B=0.25, C=0.10, D=0.05
Top-k with k=2: only consider A and B (renormalized to sum to 1),
completely IGNORE C and D no matter how sampling
proceeds from thereThis directly prevents the “occasionally picks something very unlikely and poor quality” risk from pure random sampling (Section 5) — by simply removing the least-likely options from consideration entirely, regardless of how sampling within the remaining set proceeds.
8. Top-p (Nucleus) Sampling — A More Adaptive Cutoff
Top-p sampling: include the SMALLEST set of options whose cumulative
probability reaches at least p -- an ADAPTIVE cutoff,
unlike top-k's FIXED count.
Given: A=0.60, B=0.25, C=0.10, D=0.05
Top-p with p=0.9: A (0.60) + B (0.25) = 0.85, still under 0.9, so
include C too: 0.60+0.25+0.10 = 0.95, now over 0.9
-- so the set is {A, B, C}, D is excludedThe key advantage over top-k: the number of options included adapts to how “peaked” or “flat” the distribution actually is. If the model is very confident (one option dominates), top-p naturally includes very few options. If the model is really uncertain (probabilities are spread more evenly), top-p naturally includes more options — this adapts automatically, where top-k’s fixed count doesn’t.
9. Beam Search — A Different Strategy Entirely (Brief Mention)
Worth mentioning for completeness, since it comes up in some autoregressive generation contexts: beam search doesn’t sample randomly at all — instead, it tracks several of the most promising entire sequences simultaneously as generation proceeds, ultimately selecting the overall highest-probability complete sequence among those tracked.
It’s more common in tasks like machine translation than in open-ended, creative text generation, where the more varied sampling strategies above (temperature, top-k, top-p) are generally preferred for their more natural-feeling output.
10. A Real Developer Example
Building TWO different features with the same underlying model:
Feature 1: a structured data extraction tool (pulling order numbers
from customer messages, exactly like Module 8 of the
Prompt Engineering course)
-> Use LOW temperature (near 0), or even greedy decoding --
consistency and reliability matter far more than variety for
this task.
Feature 2: a creative marketing tagline generator, producing several
DIFFERENT options for a human to choose from
-> Use HIGHER temperature (0.8-1.0) and/or top-p sampling --
variety across multiple generations is really valuable
here, and a small risk of an occasional odd option is an
acceptable trade-off since a human reviews the options anyway.This is precisely the connection to Module 26 of the Prompt Engineering course, made fully mechanical here: sampling strategy choice isn’t a stylistic afterthought — it’s a genuine engineering decision matched to what each specific task actually needs.
11. A Simple Agentic AI Connection
Sampling strategy matters directly for agent reliability. An agent generating structured tool-call parameters (Module 18 of the Prompt Engineering course) should almost always use low temperature — a tool call with a randomly “creative” but incorrect parameter value could cause a real, consequential error.
An agent’s final, user-facing conversational response might reasonably use a higher temperature for a more natural, less robotic feel. Real agent systems often deliberately vary sampling settings across different parts of a single task, exactly matching this module’s Section 10 example.
12. How Is This Used in AI?
🤖 How Is This Used in AI?
Every text, image, and audio generation API you’ll work with exposes some combination of these sampling controls (temperature, top-k, top-p) as configurable parameters. Choosing them deliberately, based on whether a task needs consistency or variety, is one of the most direct, practical levers available for tuning a generative AI application’s behavior without changing the underlying model or prompt at all.
13. Common Mistakes
Incorrect idea
Using high temperature for tasks needing consistency
Why it is incorrect
, or low temperature for tasks really benefiting from variety — as shown directly, the right setting depends entirely on the task’s actual needs, not a one-size-fits-all default.
Incorrect idea
Confusing top-k’s fixed cutoff with top-p’s adaptive cutoff.
Why it is incorrect
As demonstrated directly, top-p adjusts how many options are considered based on how confident the distribution actually is — top-k always considers exactly k options regardless of confidence.
Incorrect idea
Assuming temperature changes WHICH option is most likely.
Why it is incorrect
It doesn’t — it only reshapes how sharply the distribution favors that already-most-likely option (Section 6). The ranking of options stays the same; only the relative “closeness” of the competition changes.
14. Limitations
- No sampling strategy guarantees factually correct or high-quality output — sampling controls the shape of variability, not the underlying correctness of what the model has learned (Module 32’s hallucination discussion is a separate, related concern)
- The “right” sampling settings are really task-specific — there’s no universal correct temperature or top-p value that works best for every application
15. Quick Reference — The Whole Idea in One Diagram
Greedy: ALWAYS pick the top option -- deterministic, can feel
repetitive
Random: Sample weighted by full probability -- varied, but
can pick very unlikely, low-quality options
Temperature: reshapes the distribution's SHARPNESS before
sampling -- low = more confident/consistent,
high = more varied/exploratory
Top-k: only consider the top K options -- fixed cutoff
Top-p: only consider the SMALLEST set reaching
cumulative probability p -- ADAPTIVE cutoff16. Code — Sampling Strategies Made Concrete
🎯 Target of this example: implement and directly compare greedy decoding, temperature scaling, top-k, and top-p sampling on the exact same probability distribution — making Sections 4-8’s abstract descriptions fully concrete and numerically verifiable.
Example 1 — Simple
import numpy as np
def softmax(logits):
exp_logits = np.exp(logits - np.max(logits))
return exp_logits / np.sum(exp_logits)
# A model's predicted distribution over 4 possible next tokens
options = ["A", "B", "C", "D"]
probs = np.array([0.60, 0.25, 0.10, 0.05])
# GREEDY: always pick the single highest-probability option
greedy_choice = options[np.argmax(probs)]
print(f"Greedy decoding always picks: {greedy_choice}")
# RANDOM SAMPLING: pick according to the actual probabilities
np.random.seed(42)
random_choices = [np.random.choice(options, p=probs) for _ in range(10)]
print(f"10 random samples: {random_choices}")Expected Output:
Greedy decoding always picks: A
10 random samples: ['A', 'A', 'B', 'A', 'C', 'A', 'A', 'B', 'A', 'A']What we conclude from this example: greedy decoding produces the exact same result every single time, no variation possible. Random sampling produces genuine variety across the 10 draws — but “A” still appears most often, roughly matching its 60% probability, exactly the weighted-die intuition from Section 3.
Example 2 — Intermediate
import numpy as np
def apply_temperature(probs: np.ndarray, temperature: float) -> np.ndarray:
"""Reshapes a probability distribution by temperature -- lower
values sharpen it (favor the top option more), higher values
flatten it (make options more competitive)."""
logits = np.log(probs + 1e-10) # convert back to logit-like space
scaled_logits = logits / temperature
exp_logits = np.exp(scaled_logits - np.max(scaled_logits))
return exp_logits / np.sum(exp_logits)
options = ["A", "B", "C", "D"]
original_probs = np.array([0.60, 0.25, 0.10, 0.05])
for temp in [0.3, 1.0, 1.8]:
reshaped = apply_temperature(original_probs, temp)
print(f"Temperature {temp}: " +
", ".join(f"{o}={p:.3f}" for o, p in zip(options, reshaped)))Expected Output:
Temperature 0.3: A=0.968, B=0.030, C=0.002, D=0.000
Temperature 1.0: A=0.600, B=0.250, C=0.100, D=0.050
Temperature 1.8: A=0.421, B=0.283, C=0.183, D=0.113What we conclude from this example: at low temperature (0.3), “A” dominates almost completely (96.8%) — near-deterministic behavior. At high temperature (1.8), the gap between options narrows substantially, making B, C, and D far more competitive. This directly verifies Section 6’s claim: temperature reshapes distribution sharpness without changing which option ranks highest.
Example 3 — Production Grade
import numpy as np
def top_k_filter(probs: np.ndarray, k: int) -> np.ndarray:
"""Keep only the top K options, zero out everything else, and
renormalize so probabilities sum to 1 again."""
sorted_indices = np.argsort(probs)[::-1]
filtered = np.zeros_like(probs)
top_k_indices = sorted_indices[:k]
filtered[top_k_indices] = probs[top_k_indices]
return filtered / filtered.sum()
def top_p_filter(probs: np.ndarray, p: float) -> np.ndarray:
"""Keep the SMALLEST set of options whose cumulative probability
reaches p, zero out the rest, and renormalize -- an ADAPTIVE
cutoff, unlike top_k's fixed count."""
sorted_indices = np.argsort(probs)[::-1]
sorted_probs = probs[sorted_indices]
cumulative = np.cumsum(sorted_probs)
cutoff_idx = np.searchsorted(cumulative, p) + 1
filtered = np.zeros_like(probs)
kept_indices = sorted_indices[:cutoff_idx]
filtered[kept_indices] = probs[kept_indices]
return filtered / filtered.sum()
options = ["A", "B", "C", "D"]
probs = np.array([0.60, 0.25, 0.10, 0.05])
top_k_result = top_k_filter(probs, k=2)
top_p_result = top_p_filter(probs, p=0.9)
print("Original: ", dict(zip(options, np.round(probs, 3))))
print("Top-k (k=2): ", dict(zip(options, np.round(top_k_result, 3))))
print("Top-p (p=0.9):", dict(zip(options, np.round(top_p_result, 3))))
# Now demonstrate the ADAPTIVE nature of top-p with a FLATTER distribution
flat_probs = np.array([0.30, 0.28, 0.24, 0.18])
flat_top_p = top_p_filter(flat_probs, p=0.9)
print(f"\\nWith a FLATTER distribution {dict(zip(options, flat_probs))}:")
print("Top-p (p=0.9):", dict(zip(options, np.round(flat_top_p, 3))))Expected Output:
Original: {'A': 0.6, 'B': 0.25, 'C': 0.1, 'D': 0.05}
Top-k (k=2): {'A': 0.706, 'B': 0.294, 'C': 0.0, 'D': 0.0}
Top-p (p=0.9): {'A': 0.632, 'B': 0.263, 'C': 0.105, 'D': 0.0}
With a FLATTER distribution {'A': 0.3, 'B': 0.28, 'C': 0.24, 'D': 0.18}:
Top-p (p=0.9): {'A': 0.3, 'B': 0.28, 'C': 0.24, 'D': 0.18}What we conclude from this example: on the original, peaked distribution, top-p included 3 options (A, B, C) to reach 90% cumulative probability. On the flatter distribution, top-p had to include ALL 4 options just to reach 90% — this is Section 8’s adaptive behavior directly verified: top-p automatically adjusts how many options it considers based on how confident or uncertain the underlying distribution actually is, something top-k’s fixed count cannot do.
17. Interview Questions
Q: What does it mean to “sample” from a model’s predicted probability distribution?
Ans: It means selecting one concrete output where the choice is influenced by the predicted probabilities — options with higher predicted probability are more likely to be selected, but it’s not a guarantee like always picking the top option. This is fundamentally different from greedy decoding, which deterministically always selects the single highest-probability option with no randomness at all.
Q: How does temperature affect a model’s output, and what’s a common misconception about what it changes?
Ans: Temperature reshapes the sharpness of the probability distribution before sampling — low temperature makes the distribution more sharply concentrated on the already-most-likely option (closer to deterministic behavior), while high temperature flattens the distribution, making lower-probability options relatively more competitive. A common misconception is that temperature changes WHICH option is ranked highest — it doesn’t; it only changes how dominant that top option is relative to the others.
Q: What’s the key difference between top-k and top-p sampling?
Ans: Top-k always considers exactly k highest-probability options, regardless of how confident or spread out the underlying distribution actually is — a fixed cutoff. Top-p (nucleus sampling) instead includes the smallest set of options whose cumulative probability reaches a threshold p — an adaptive cutoff that naturally includes fewer options when the model is confident (one option dominates) and more options when the model is really uncertain (probabilities are more spread out).
Q: How would you decide on sampling settings for two different features built on the same underlying model: a structured data extraction tool and a creative tagline generator?
Ans: For the structured extraction tool, I’d use low temperature (or even greedy decoding), since consistency and reliability matter far more than variety — the same input should reliably produce the same correct extraction. For the creative tagline generator, I’d use higher temperature and/or top-p sampling, since generating several really different options is valuable, and a human reviewing the results makes the small risk of an occasional lower-quality option an acceptable trade-off for increased creative variety.
18. What You Should Remember
- Sampling converts a model’s probability distribution into one concrete output — really different strategies (greedy, random, temperature-adjusted, top-k, top-p) produce meaningfully different behavior.
- Temperature reshapes distribution sharpness, not ranking — low temperature for consistency, high temperature for variety, verified directly with numerical reshaping.
- Top-p adapts to distribution confidence, unlike top-k’s fixed cutoff — verified directly by comparing behavior on a peaked vs. a flat distribution.
19. Quick Practice
For each of these tasks, decide on an appropriate sampling approach (greedy, low temperature, high temperature, top-k, or top-p) and justify your choice: (1) generating a legal document’s boilerplate text, (2) brainstorming 10 different startup name ideas, (3) translating a sentence into French.
20. Next Step
Next: Module 11 — Latent Space — the concept you first encountered in Module 7’s VAE discussion, now covered in its own right as a foundational idea that reappears throughout Generative AI, including directly in diffusion models.
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed