Temperature, Top-K and Top-P

Before you continue: three tools for this module

Inference: using fixed learned parameters to produce output.
Logit: a raw score for a possible next token.
Decoding: the rule that chooses tokens from model scores.

You do not need to memorize these yet. Use this map when the terms reappear.

Begin with the central question

What hidden problem does Temperature, Top-K and Top-P solve inside a real language-model system?

Keep that central question about Temperature, Top-K and Top-P in mind. The definitions, numbers, diagrams, and code examples below answer it one piece at a time.

probabilities → filtering/reshaping → token selection

1. What You Will Learn

Learning outcomes

Explain how temperature reshapes a probability distribution.
Compare top-k filtering with top-p nucleus sampling.
Predict how decoding settings affect diversity and repeatability.
Choose conservative or creative settings based on application risk.

In one sentence

💡 Big picture

Temperature, top-k, and top-p change how the model chooses among possible next tokens; they do not add new knowledge.

2. Why This Module Exists

The problem this module solves

Always choosing the top token can be repetitive, while sampling too freely can create nonsense.
Different tasks need different levels of variety and predictability.

3. Intuition

the model always produces the same kind of output — a probability distribution over the vocabulary (Module 5). Sampling strategies decide how to actually pick one token from that distribution: always the top pick (greedy), reshaped to be more or less “peaked” (temperature), restricted to only the most likely options (top-k), or restricted to just enough options to cover a target probability mass (top-p).

4. Core Concept

Logits
 ↓
Probability distribution     (Module 5's softmax)
 ↓
Token selection

Term	Definition
Greedy decoding	Always select the single highest-probability token
Temperature	Rescales logits before softmax — lower makes the distribution sharper (more confident); higher makes it flatter (more random)
Top-K	Restrict selection to only the K highest-probability tokens, renormalize, then sample
Top-P (nucleus sampling)	Restrict selection to the smallest set of tokens whose cumulative probability reaches P, renormalize, then sample
Beam search	Tracks multiple candidate sequences simultaneously, rather than committing to one token at a time (used less commonly for open-ended LLM generation)
Random sampling	Draw a token according to the (possibly reshaped/filtered) probability distribution, rather than deterministically picking the top one

5. How It Works — Step by Step

Temperature, precisely:

scaled_logits = logits / T

T < 1:   makes DIFFERENCES between logits LARGER after softmax
         -> SHARPER, more confident distribution
T = 1:   the original, unmodified distribution
T > 1:   makes differences SMALLER -> FLATTER, more random
         distribution

Top-K: keep only the K highest-logit tokens, set every other token’s probability to zero (via -infinity logit before softmax), renormalize, then sample from this restricted set.

Top-P: sort tokens by probability descending, keep adding tokens until their cumulative probability reaches the threshold P, discard the rest, renormalize, then sample.

6. Mathematical Intuition

Read the mathematics as a story

probabilities → filtering/reshaping → token selection

First locate the input, operation, and output. Then treat the formula as a compact description of that journey rather than a collection of symbols to memorize.

Entropy (a genuine measure of a distribution’s “randomness” or uncertainty) directly captures temperature’s effect: lower temperature produces lower entropy (more predictable, concentrated distribution); higher temperature produces higher entropy (more spread out, unpredictable distribution) — verified directly below.

Analogy: The Coffee Shop Order Board & Gaffer Tape Filters Think of choosing a token from a probability distribution like picking a coffee flavor from a board:

The Setup: The barista has a list of 8 options with their popularity percentages: “Vanilla (47%)”, “Caramel (31%)”, “Mocha (8%)”, and so on.

Greedy Selection (The Regular): You are boring; you always buy the single top-seller (“Vanilla”).

Temperature (The Mood Swings):

Low Temperature (T = 0.3): The board is heavily biased. The top flavor gets boosted to 78%, and all others shrink. You almost always pick “Vanilla”.

High Temperature (T = 2.0): The board is flattened. Vanilla drops to 30%, and less common flavors like lavender get boosted. You get creative.

Top-K (The Barista’s Tape): The barista covers all but the top 3 items with black tape. You are forced to choose randomly among only those three.

Top-P (The Crowd-Pleaser Tape): The barista adds up the percentages from most popular to least until they reach 90% (the “nucleus”). Everything else gets taped over. If the crowd is highly confident, only 2 flavors stay exposed. If the crowd is uncertain, 6 flavors stay exposed.

📊 Visual Flowchart: The Logits Sampling Pipeline

Here is how raw vocabulary scores are transformed and pruned before selection:

graph TD
    classDef raw fill:#e74c3c,stroke:#333,stroke-width:1px,color:#fff;
    classDef temp fill:#3498db,stroke:#333,stroke-width:1px,color:#fff;
    classDef prune fill:#f1c40f,stroke:#333,stroke-width:1px,color:#fff;
    classDef select fill:#2ecc71,stroke:#333,stroke-width:1px,color:#fff;

    Logits["Raw Logits:<br>['blue': 4.2, 'clear': 3.8, ... 'purple': -1.5]"]:::raw --> TempDiv["1. Temperature Scaling:<br>scaled_logits = logits / T"]

    TempDiv --> Softmax["2. Softmax Normalization:<br>base_probs = softmax(scaled_logits)"]:::temp

    subgraph Filters ["Distribution Pruning (Top-K / Top-P)"]
        Softmax --> TopK["3a. Top-K Filter:<br>Keep highest K tokens, zero out others"]:::prune
        Softmax --> TopP["3b. Top-P Filter:<br>Keep smallest subset covering P cumulative mass, zero others"]:::prune
    end

    TopK --> Renorm["4. Renormalize remaining probabilities (Sum to 1.0)"]
    TopP --> Renorm

    Renorm --> RandomSelect["5. Random Sampling draw from final distribution"]:::select
    RandomSelect --> NextToken["6. Output chosen token"]

7. Small Worked Example

Walk through the example

Name what each input represents.
Follow one transformation at a time.
Translate the result back into ordinary language.

The purpose is to reveal the mechanism, not merely display an answer.

For “the sky is ___,” a low temperature (e.g., 0.3) would make the model strongly favor its single most likely continuation almost every time. A high temperature (e.g., 2.0) would flatten the distribution, giving less-likely words a meaningfully higher chance of being selected — more creative, but also more likely to produce implausible output.

8. Python Example

What the code will demonstrate

The code builds a tiny version of the mechanism, prints values you can inspect, and connects them to the worked example. Predict the direction of the result before running it.

Python symbols used below

NumPy (np) stores numeric vectors and matrices.
np.array(...) creates a numeric collection.
Library calls perform the same conceptual steps shown above at a larger scale.

# Build a small, inspectable example of Temperature, Top-K and Top-P.
# Follow the inputs, transformations, and output in order.
import numpy as np

def softmax(x):
    exp_x = np.exp(x - np.max(x))
    return exp_x / np.sum(exp_x)

vocab = ["blue", "clear", "dark", "cloudy", "green", "purple", "gray", "bright"]
logits = np.array([4.2, 3.8, 2.1, 2.5, 0.3, -1.5, 1.8, 0.9])

base_probs = softmax(logits)
print("Base distribution (temperature=1.0):")
for w, p in sorted(zip(vocab, base_probs), key=lambda x: -x[1]):
    print(f"  {w:8s}: {p:.4f}")

# --- Temperature ---
def apply_temperature(logits, T):
    return softmax(logits / T)

print("\n--- TEMPERATURE ---")
for T in [0.3, 1.0, 2.0]:
    probs = apply_temperature(logits, T)
    entropy = -np.sum(probs * np.log(probs + 1e-10))
    print(f"T={T}: top token='{vocab[np.argmax(probs)]}' (p={probs.max():.4f}), entropy={entropy:.4f}")

# --- Top-K ---
def top_k_filter(logits, k):
    top_k_idx = np.argsort(logits)[-k:]
    filtered = np.full_like(logits, -np.inf)
    filtered[top_k_idx] = logits[top_k_idx]
    return softmax(filtered)

print("\n--- TOP-K ---")
for k in [2, 4, 8]:
    probs = top_k_filter(logits, k)
    nonzero = np.sum(probs > 1e-6)
    print(f"K={k}: {nonzero} tokens nonzero, top: {sorted(zip(vocab, probs), key=lambda x: -x[1])[0]}")

# --- Top-P (nucleus sampling) ---
def top_p_filter(logits, p_threshold):
    probs = softmax(logits)
    sorted_idx = np.argsort(probs)[::-1]
    cumulative = np.cumsum(probs[sorted_idx])
    cutoff = np.searchsorted(cumulative, p_threshold) + 1
    keep_idx = sorted_idx[:cutoff]
    filtered = np.full_like(logits, -np.inf)
    filtered[keep_idx] = logits[keep_idx]
    return softmax(filtered), cutoff

print("\n--- TOP-P ---")
for p_thresh in [0.5, 0.9]:
    probs, cutoff = top_p_filter(logits, p_thresh)
    print(f"P={p_thresh}: kept {cutoff} tokens covering {p_thresh*100:.0f}% probability mass")

Expected Output:

Base distribution (temperature=1.0):
  blue    : 0.4702
  clear   : 0.3152
  cloudy  : 0.0859
  dark    : 0.0576
  gray    : 0.0427
  bright  : 0.0173
  green   : 0.0095
  purple  : 0.0016

--- TEMPERATURE ---
T=0.3: top token='blue' (p=0.7884), entropy=0.5376
T=1.0: top token='blue' (p=0.4702), entropy=1.3533
T=2.0: top token='blue' (p=0.3040), entropy=1.8033

--- TOP-K ---
K=2: 2 tokens nonzero, top: ('blue', 0.5987...)
K=4: 4 tokens nonzero, top: ('blue', 0.5062...)
K=8: 8 tokens nonzero, top: ('blue', 0.4702...)

--- TOP-P ---
P=0.5: kept 2 tokens covering 50% probability mass
P=0.9: kept 4 tokens covering 90% probability mass

9. How It Works

Temperature’s effect on entropy is monotonic and direct: T=0.3 gives entropy 0.5376 (sharp, confident — “blue” gets 78.8% probability), T=1.0 gives 1.3533 (moderate), T=2.0 gives 1.8033 (flat, uncertain — “blue” drops to 30.4%). Lower temperature genuinely produces a more confident, deterministic-feeling distribution; higher temperature genuinely produces a flatter, more exploratory one.
Top-K restricts the candidate set to exactly K tokens, regardless of how much probability mass they represent — K=2 keeps only “blue” and “clear,” renormalized to 59.87%/40.13%.
Top-P instead restricts based on cumulative probability, adapting the number of kept tokens to the distribution’s actual shape — P=0.5 happened to need only 2 tokens here (since “blue” and “clear” alone already exceed 50%), while P=0.9 needed 4. This adaptivity is precisely top-p’s advantage over top-k: it naturally uses fewer candidates when the distribution is confident/peaked, and more when it’s genuinely uncertain/flat.

10. How Is This Used in Modern AI?

Trace it through a real model call

user message → assembled context → LLM computation → decoded output → application checks

This topic affects one stage of that path; it is not the complete product. Hosted GPT- and Gemini-style applications also add instructions, safety systems, retrieval, tools, serving infrastructure, and evaluation around the model.

🤖 How Is This Used in Modern AI?

Every LLM API exposes these exact parameters (commonly temperature, top_p, sometimes top_k) — verified directly, they operate on the real underlying probability distribution, not some separate mechanism. Understanding them precisely is directly, practically useful for tuning API behavior.

Setting	Typical use case
Low temperature (near 0)	Factual Q&A, code generation, tasks needing consistency
Higher temperature	Creative writing, brainstorming, more varied output
Top-p (commonly used)	A robust default that adapts to the distribution’s actual shape

11. How Is This Used in Agentic AI?

Separate the model from the runtime

goal + state + tool results → LLM proposal → runtime validation → execution or response

The LLM proposes text or a structured action. Ordinary application code controls permissions, tools, retries, memory, and execution.

Direct relevance to Agentic AI: High, practically. Agent applications often deliberately use low temperature (or greedy decoding) for tool-call generation and structured output — consistency and correctness matter more than creative variation for a JSON tool call. Higher temperature might be appropriate for a conversational or brainstorming-oriented agent response.

12. Common Beginner Mistakes

⚠️ Mistake

Incorrect idea: assuming these settings change what the model “knows.”

Why it is incorrect: They don’t — they operate purely on the SAME underlying probability distribution (Module 5), reshaping how a token is selected from it, not changing the distribution’s fundamental content or the model’s weights.

⚠️ Mistake

Incorrect idea: assuming higher temperature always improves output quality/creativity.

Why it is incorrect: As shown directly, higher temperature increases randomness (entropy) — this can help avoid repetition (Module 7’s demonstrated failure mode) but can also increase the chance of selecting implausible, lower-quality tokens.

⚠️ Mistake

Incorrect idea: confusing top-k and top-p.

Why it is incorrect: Top-k always keeps exactly K tokens regardless of their combined probability; top-p adaptively keeps however many tokens are needed to reach a target cumulative probability — genuinely different mechanisms, verified directly with different behaviors on the same distribution.

13. Important Distinctions

Greedy Decoding	Sampling-Based Decoding
Always picks the single highest-probability token	Introduces controlled randomness
Fully deterministic	Can vary between runs (unless seeded)

Top-K	Top-P
Fixed NUMBER of candidate tokens	Fixed cumulative PROBABILITY threshold — adapts token count to distribution shape

Temperature	Top-K / Top-P
Reshapes the ENTIRE distribution	Restricts to a SUBSET, then samples from it

14. When to Use

Use low temperature or greedy decoding for tasks needing consistency and correctness (structured output, code, factual answers). Use moderate-to-high temperature with top-p for more varied, creative generation. Top-p is a commonly-used, robust default across many applications, given its adaptivity to distribution shape.

15. When Not to Use

Avoid high temperature for tasks requiring precise, structured, or deterministic output — the increased randomness (verified directly via entropy) genuinely increases the chance of selecting implausible tokens, directly undesirable for tool calls or factual precision.

16. Production Considerations

Temperature and top-p/top-k are genuinely configurable per API request — a real, practical lever for tuning application behavior without any model retraining.
Reproducibility requires care — sampling-based decoding introduces genuine randomness; applications needing reproducible output should use greedy decoding (or a fixed random seed, where supported) rather than high-temperature sampling.
Repetition penalties (Module 7’s demonstrated failure mode) are often combined with these sampling strategies in production systems.

17. What You Should Remember

Every sampling strategy operates on the exact same underlying probability distribution (Module 5) — nothing about the model’s actual knowledge or weights changes.
Temperature reshapes the entire distribution — verified directly: lower temperature sharply increases confidence (lower entropy); higher temperature flattens it (higher entropy).
Top-K keeps a fixed number of candidates; Top-P adaptively keeps enough candidates to reach a target cumulative probability — verified directly with genuinely different behaviors on the same distribution.

18. Interview Questions

Beginner

Q: What does temperature do during LLM text generation?

Ans: It rescales the model’s logits before softmax — lower temperature sharpens the resulting probability distribution (making the model more confident/deterministic in its top choice), while higher temperature flattens it (making the selection more random and varied).

Intermediate

Q: What’s the difference between top-k and top-p sampling?

Ans: Top-k restricts token selection to a fixed NUMBER (K) of the highest-probability tokens, regardless of how much total probability mass they represent.

Top-p (nucleus sampling) instead restricts selection to the SMALLEST set of tokens whose cumulative probability reaches a target threshold P — verified directly, this means top-p adaptively uses fewer tokens when the distribution is confident/peaked and more tokens when it’s flatter/more uncertain, while top-k always uses exactly K regardless of the distribution’s actual shape.

Advanced

Q: Why does temperature affect randomness, mathematically?

Ans: Temperature divides logits by T before applying softmax. Since softmax exponentiates its inputs, dividing logits by a value less than 1 (low temperature) amplifies the relative differences between them before exponentiation, producing a sharper, more concentrated probability distribution after normalization.

Dividing by a value greater than 1 (high temperature) compresses these relative differences, producing a flatter distribution.

This was verified directly via entropy: lower temperature (0.3) produced entropy of 0.5376 (a sharp, confident distribution), while higher temperature (2.0) produced entropy of 1.8033 (a much flatter, more uncertain distribution) — entropy being a direct mathematical measure of a distribution’s uncertainty/ randomness.

Scenario

Q: A team building a code-generation assistant notices occasional syntactically invalid output when using a moderate-to-high temperature setting. What would you recommend, and why?

Ans: I’d recommend lowering the temperature significantly, or switching to greedy/near-greedy decoding, for this use case.

Code generation generally benefits from consistency and precision rather than creative variation — as demonstrated directly, higher temperature increases the probability of selecting lower-probability (and potentially syntactically implausible) tokens, which is a genuine, direct explanation for occasional invalid output.

Structured, precision- sensitive tasks like code generation are typically better served by low temperature, keeping the model close to its highest-confidence predictions.

AI Engineering

Q: Why might a production system use top-p rather than top-k as its primary sampling restriction strategy?

Ans: Top-p adapts to the actual shape of the model’s probability distribution at each specific generation step — verified directly, it kept only 2 tokens when the distribution was confident/peaked (P=0.5) but 4 tokens when more coverage was needed (P=0.9), automatically adjusting based on how concentrated or spread out the model’s confidence actually is at that moment.

Top-k, by contrast, always considers exactly K tokens regardless of whether the model was highly confident (where a large K might unnecessarily include very unlikely tokens) or genuinely uncertain (where a small K might exclude reasonable alternatives) — top-p’s adaptivity is a genuine, practical advantage that’s part of why it’s commonly used as a default or combined with top-k in production systems.

19. Next Step

Next: Module 16 — Fine-Tuning — why it exists, when it’s the right choice versus RAG, and how it builds on the pretrained weights this course has now completely traced from creation to inference.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026