How GPT-Style LLMs Actually Work

Begin with the central question

What chain of computations turns a prompt into one next-token probability distribution?

Essential words

A logit is an unnormalized score for a vocabulary token. A probability distribution comes from softmax. Autoregressive generation appends one selected token and repeats.

What You Will Understand

This is a major integration module. You’ll trace one real request — “Explain RAG” — completely through a GPT-style decoder-only model: tokenization, embeddings, positional information, causally-masked Transformer blocks, the LM head, logits, softmax, and a predicted next token. Nothing new is introduced — every piece is something you’ve already built and verified in Modules 2-13.

prompt -> tokens -> decoder blocks -> logits -> sampling -> append token -> repeat

The problem this module solves

Every module so far examined one piece in isolation. This module exists to assemble them into the single thing you actually came to understand: what genuinely happens when you send a prompt to a real LLM. By the end, “LLM generation is fundamentally repeated next-token prediction” should feel mechanically obvious, not like a slogan to memorize.

Build the intuition

an LLM is a very large function that takes in a sequence of tokens and outputs exactly one thing: a probability distribution over “what token comes next.” Generating a full response is this one operation, repeated — feed in everything so far, get a next-token distribution, pick one, append it, repeat.

Analogy: The Autoregressive Conveyor Belt Loop Think of next-token text generation in GPT-style models as a manufacturing assembly line that feeds its own output back into the raw material hopper:

The Setup: You are manufacturing a chain link. You load the initial blueprint link “Explain” and “RAG” onto the conveyor belt.

The Analysis: The belt rolls through three repeating floors of blocks. The final inspector looks only at the last link (“RAG”) and queries a dictionary list (LM head) of vocabulary words to decide the next link probability.

The Output: The selector stamps the next link “is”.

The Loop: Instead of finishing, a mechanical arm picks up “is”, physically welds it to the end of the input belt, making the new sequence “Explain RAG is”, and pushes it back to the very front of the factory.

The factory restarts. The next link is computed based on this expanded context, generating “a”, which is welded on, restarting again to generate “retrieval”.

📊 Visual Flowchart: Autoregressive Next-Token Generation Loop

Here is the step-by-step cycle of feeding generated outputs back into the input sequence:

graph TD
    classDef process fill:#2ecc71,stroke:#333,stroke-width:1px,color:#fff;
    classDef loop fill:#e74c3c,stroke:#333,stroke-width:1px,color:#fff;




    Prompt["Input Sequence:<br>'Explain RAG'"] --> Tokenizer["1. Tokenizer & Embedding Lookup"]
    Tokenizer --> Blocks["2. Stack of Causal Decoder Blocks"]:::process




    Blocks --> LMHead["3. LM Head Projects Last Position to Logits"]
    LMHead --> Softmax["4. Softmax Probability Distribution"]
    Softmax --> Select["5. Token Selection (Greedy/Argmax)<br>Outputs: 'is'"]:::process




    Select --> AppendLoop{"6. Append Output Token<br>Is Next Token '<eos>'?"}:::loop




    AppendLoop -->|No| AppendSeq["New Sequence:<br>'Explain RAG is'"]
    AppendSeq --> Tokenizer




    AppendLoop -->|Yes| Finish["Generation Complete!"]:::process

4. Core Concept — The Complete Trace

"Explain RAG"
  ↓
Tokenizer                    (Module 2)
  ↓
Token IDs                     (Module 2)
  ↓
Embeddings                     (Module 2)
  ↓
Positional information           (Module 8)
  ↓
Transformer Block 1               (Module 9 — CAUSAL self-attention,
                                  Module 6 — plus residuals, norm, FFN)
  ↓
Transformer Block 2
  ↓
...
  ↓
Final hidden representation          (Module 12)
  ↓
LM Head                                (a linear projection to
                                       vocabulary size — same
                                       mechanism as any linear
                                       layer, DL Module 2)
  ↓
Logits
  ↓
Softmax                                  (Module 4 of this course /
                                         DL Module 4)
  ↓
Token probabilities
  ↓
Next token

Then: the new token is appended to the sequence, and the entire process repeats to generate the token after that.

5. How It Works — Step by Step

1. Tokenize the prompt into tokens, then token IDs (Module 2)
2. Look up each token ID's embedding, add positional information
   (Module 2, 8)
3. Pass the resulting sequence through EVERY decoder block in the
   model, in order -- each block applying CAUSAL self-attention
   (Module 6) + residual/norm (Module 11) + FFN (Module 10),
   exactly as assembled in Module 9 and stacked in Module 12
4. The FINAL block's output is a hidden representation for EVERY
   position -- but only the LAST position's representation is
   used to predict the next token
5. The LM HEAD projects this last position's hidden state to a
   vector the size of the entire VOCABULARY -- these raw scores
   are LOGITS
6. SOFTMAX converts the logits into a genuine probability
   distribution
7. A token is SELECTED (e.g., greedily, the highest-probability
   token -- more sophisticated sampling strategies exist but are
   beyond this course's scope)
8. The selected token is APPENDED to the sequence
9. Return to step 3 -- repeat for the NEXT token, now with one
   more token of context than before

6. Mathematical Intuition

Read the mathematics as a story

A GPT-style generation step ends with one vector of vocabulary logits. Softmax turns them into probabilities; a decoding rule selects one token; the token is appended and the process repeats.

prompt -> decoder blocks -> last-position state -> logits -> probabilities -> one token -> repeat

Nothing new — this module’s entire value is watching Modules 2, 6, 8, 9, and 12’s individually-verified computations connect into one continuous pipeline, traced below with real numbers on a real (tiny) example.

7. Small Worked Example

Walk through the example

Process a tiny prompt. 2. take the last position state. 3. Project it to vocabulary logits. 4. Convert to probabilities. 5. Select and append a token.

Given the 2-token prompt “Explain RAG,” a GPT-style model processes both tokens through its full stack of causally-masked decoder blocks. Only the second token’s (“RAG”) final hidden state — since it’s the last position in the sequence — gets projected to logits and used to predict what comes next.

The model has never seen the words “is,” “a,” or “retrieval” that might plausibly follow, in this tiny untrained example — but the same mechanism, at real scale with real trained weights, is exactly what produces coherent continuations from actual LLMs.

8. Python / NumPy Example

What the code will demonstrate

This small NumPy example makes How GPT-Style LLMs Actually Work visible with inspectable numbers and shapes. Read it in three passes: identify each input, follow the transformation line by line, and connect the printed output to the diagram above. The arrays are intentionally tiny teaching values; unless the text explicitly says otherwise, they are not weights or measurements from GPT, Gemini, or another trained model.

# The arrays are intentionally small so each transformation can be inspected.
# Printed values illustrate the mechanism; they are not trained-model measurements.
import numpy as np




def softmax(x, axis=-1):
    exp_x = np.exp(x - np.max(x, axis=axis, keepdims=True))
    return exp_x / np.sum(exp_x, axis=axis, keepdims=True)




def layer_norm(x, eps=1e-8):
    mean = x.mean(axis=-1, keepdims=True)
    std = x.std(axis=-1, keepdims=True)
    return (x - mean) / (std + eps)




def relu(x): return np.maximum(0, x)




def causal_mask(seq_len):
    return np.triu(np.ones((seq_len, seq_len)), k=1).astype(bool)




np.random.seed(14)




# --- Tokenizer -> Token IDs ---
vocab = ["Explain", "RAG", "is", "a", "retrieval", "method", "<eos>"]
prompt_tokens = ["Explain", "RAG"]
token_ids = [vocab.index(t) for t in prompt_tokens]
print("Prompt:", " ".join(prompt_tokens))
print("Token IDs:", token_ids)




d_model = 8
vocab_size = len(vocab)




# --- Embeddings + Positional information ---
embedding_matrix = np.round(np.random.randn(vocab_size, d_model) * 0.4, 3)
token_embeds = embedding_matrix[token_ids]




def positional_encoding(seq_len, d_model):
    pos = np.arange(seq_len)[:, np.newaxis]
    i = np.arange(d_model)[np.newaxis, :]
    angle_rates = 1 / np.power(10000, (2 * (i // 2)) / np.float32(d_model))
    angles = pos * angle_rates
    pe = np.zeros((seq_len, d_model))
    pe[:, 0::2] = np.sin(angles[:, 0::2])
    pe[:, 1::2] = np.cos(angles[:, 1::2])
    return pe




x = token_embeds + positional_encoding(len(prompt_tokens), d_model)
print("\nTransformer input shape:", x.shape)




# --- Stack of CAUSAL decoder blocks ---
def decoder_block(x, seed):
    rng = np.random.RandomState(seed)
    seq_len = x.shape[0]
    Wq, Wk, Wv, Wo = [rng.randn(d_model, d_model) * 0.3 for _ in range(4)]
    Q, K, V = x @ Wq, x @ Wk, x @ Wv
    scores = Q @ K.T / np.sqrt(d_model)
    scores[causal_mask(seq_len)] = -np.inf   # CAUSAL masking (Module 6)
    attn_out = (softmax(scores, axis=-1) @ V) @ Wo
    x = layer_norm(x + attn_out)




    d_ff = d_model * 2
    W1 = rng.randn(d_model, d_ff) * 0.3
    W2 = rng.randn(d_ff, d_model) * 0.3
    ffn_out = relu(x @ W1) @ W2
    x = layer_norm(x + ffn_out)
    return x




num_layers = 3
for layer_idx in range(num_layers):
    x = decoder_block(x, seed=layer_idx)




print("Final hidden representation:\n", np.round(x, 4))




# --- LM Head: project LAST position's hidden state to vocab size ---
last_hidden = x[-1]
W_lm_head = np.round(np.random.randn(vocab_size, d_model) * 0.4, 2)
logits = W_lm_head @ last_hidden
print("\nLogits:")
for word, logit in zip(vocab, logits):
    print(f"  {word:12s}: {logit:.4f}")




# --- Softmax -> probabilities ---
probs = softmax(logits)
print("\nProbability distribution over next token:")
for word, prob in zip(vocab, probs):
    print(f"  {word:12s}: {prob:.4f}")
print("Sum:", probs.sum())




next_token_id = np.argmax(probs)
print(f"\nPredicted next token: '{vocab[next_token_id]}'")

Expected Output:

Prompt: Explain RAG
Token IDs: [0, 1]




Transformer input shape: (2, 8)
Final hidden representation:
 [[-0.7682 -0.2214 -0.1182  1.0726 -1.6504 -0.6181  1.6193  0.6844]
 [-0.6471 -0.1905 -0.0413  0.9344 -1.7325 -0.5531  1.7769  0.4532]]




Logits:
  Explain     : 0.6765
  RAG         : -1.7306
  is          : -1.0857
  a           : 1.3636
  retrieval   : -0.2210
  method      : -0.6975
  <eos>       : -0.4073




Probability distribution over next token:
  Explain     : 0.2354
  RAG         : 0.0212
  is          : 0.0404
  a           : 0.4679
  retrieval   : 0.0959
  method      : 0.0596
  <eos>       : 0.0796
Sum: 1.0




Predicted next token: 'a'

9. How It Works

Only the last position’s hidden state (x[-1], corresponding to “RAG”) is used for prediction — the first position’s (“Explain”) hidden state was computed too, but isn’t used for this specific next-token prediction. In a real generation loop, every position’s next-token prediction is computed and used during training (Module 15), but during inference, only the final position’s prediction determines what token comes next.
The probability distribution genuinely sums to 1.0 — softmax’s defining property, confirmed once more at the scale of a real (if tiny) vocabulary within a complete pipeline.
This untrained toy model predicts “a” as most likely (0.468) — with random weights, this has no real linguistic meaning, but the mechanism — causal decoder blocks → last hidden state → LM head → logits → softmax → selection — is exactly what a real, trained LLM runs, just with weights that have been optimized (Module 15) to make linguistically sensible predictions rather than random ones.

10. How Is This Used in Modern AI?

Where this concept lives

Follow the concept at three levels: inside the model, where the computation happens; inside the AI product, where that computation supports a visible feature; and inside production, where engineers measure speed, memory, quality, and failure cases. The details below connect those levels.

🤖 How Is This Used in Modern AI?

This entire trace is what happens, mechanically, every time you send a prompt to Claude, GPT, or any other decoder-only LLM. Nothing more mysterious happens — just this pipeline, run at a scale of thousands of dimensions, dozens to over a hundred layers, and a vocabulary of tens of thousands of tokens, with weights trained on enormous amounts of text (Module 15).

11. How Is This Used in LLMs?

Trace one model call

User text → tokens → Transformer computation → output-token probabilities
          this topic affects one part of that computation

An LLM does not apply this idea as a separate magic step. It uses it as part of the repeated numerical pipeline that transforms token vectors and produces the next-token probabilities.

Generating a full multi-token response means repeating this entire trace once per output token — each time with one more token in the context than before (the previously generated token gets appended). This repeated-forward-pass structure is directly why longer generations take more time, and is exactly the mechanism Module 16’s KV cache exists to make more efficient.

Real systems you can recognize

GPT-style models generate autoregressively with causal decoder blocks. Gemini’s public API also reports tokenized inputs and generated outputs, while its full proprietary internal architecture is not completely disclosed; avoid claiming undocumented details.

12. How Is This Used in Agentic AI?

Trace one agent step

Goal + history + tool results
              ↓
     LLM processes the context
              ↓
Suggested answer or tool call
              ↓
Agent runtime validates and executes it

This distinction matters: the Transformer helps produce the proposal, while the surrounding agent software controls tools, permissions, retries, memory, and execution.

Direct relevance to Agentic AI: Very High. Every reasoning step an agent’s LLM performs — deciding which tool to call, interpreting a result, drafting a response — is one or more runs of exactly this trace.

An agent’s “context” is literally the token sequence fed into step 1 of this pipeline at each point; understanding this mechanism is what makes an agent’s behavior explicable as a direct, traceable consequence of what tokens were in context, not an unexplainable black box.

When this knowledge is useful

Use How GPT-Style LLMs Actually Work when you need to explain, implement, debug, evaluate, or optimize the corresponding part of a Transformer pipeline. It is also useful when a model API behaves unexpectedly and you need to trace the behavior back to tokens, tensor shapes, attention visibility, training, or inference mechanics.

When it is not enough

Understanding this mechanism does not by itself prove that a complete model or application is accurate, safe, fast, or cost-effective. Production decisions still require representative evaluation data, latency and memory measurements, model-specific documentation, and tests of the surrounding retrieval or agent code.

13. Common Beginner Mistakes

⚠️ Mistake

Incorrect idea: assuming the model “considers” the whole vocabulary meaningfully differently for each choice.

Why it is incorrect: Every generation step runs the exact same mechanism — compute logits for the entire vocabulary, softmax, select — there’s no special-cased reasoning process separate from this repeated computation.

⚠️ Mistake

Incorrect idea: thinking every token position’s hidden state matters equally for the next prediction.

Why it is incorrect: As demonstrated, only the last position’s hidden state is used to predict what comes after it during inference — earlier positions’ hidden states matter because they contributed, via attention, to shaping later positions’ hidden states, not because they’re directly used for this specific prediction.

⚠️ Mistake

Incorrect idea: believing “LLM generation is repeated next-token prediction” is just a slogan.

Why it is incorrect: As traced concretely here, it’s a literal, mechanical description of exactly what computation runs, step by step, for every single token generated.

14. Important Distinctions

Logits	Probabilities
Raw, unbounded output of the LM head	After softmax — bounded (0,1), sum to 1

Hidden Representation (all positions)	Next-Token Prediction Input
Computed for EVERY token position	Only the LAST position is used to predict the next token during inference

15. Production / Engineering Considerations

Every generated token requires a full forward pass through the entire stack of decoder blocks — this repeated computation is the direct source of generation latency, addressed practically by KV caching (Module 16).
The LM head’s parameter count (vocab_size × d_model) mirrors the embedding matrix’s size (Module 2) — in many models, these two weight matrices are even shared (“tied weights”), a practical parameter- saving technique worth knowing exists, though implementation details vary by model.

16. Interview Questions

Beginner

Q: What does an LLM actually produce when it processes a prompt?

Ans: A probability distribution over its entire vocabulary, representing its prediction for what token should come next, given everything in the current context. Generating a full response means repeating this — predict a next-token distribution, select a token, append it, repeat.

Intermediate

Q: Why does the model use only the LAST token position’s hidden state to predict the next token, even though every position has one?

Ans: Next-token prediction is specifically asking “what comes after everything so far” — the last position’s hidden state, having been built through causal self-attention (Module 6) over the entire preceding sequence, is the representation that has “seen” all the relevant context for that specific prediction.

Earlier positions’ hidden states remain important because they shaped, via attention, what the last position’s representation ended up containing — but they aren’t directly used for this specific next-token prediction during inference.

Advanced

Q: Trace exactly what happens, mechanically, between a model receiving a prompt and producing its very first output token.

Ans: The prompt is tokenized and converted to token IDs, each looked up in the embedding matrix and combined with positional information. This sequence passes through every decoder block in the model — each applying causally-masked self-attention (so no position can see later positions), residual connections, layer normalization, and a feed-forward network. The final block’s output includes a hidden representation for every position, but only the last position’s representation is projected by the LM head into logits over the entire vocabulary.

Softmax converts these logits into a probability distribution, from which a token is selected — that selected token is the model’s first output token.

Scenario

Q: A user sends a 500-token prompt and receives a 50-token response. How many full forward passes through the model’s decoder blocks happened, roughly, to produce that response?

Ans: Roughly 50 — one full forward pass per generated token (in the most basic, uncached formulation). Each pass processes the growing sequence (prompt plus all tokens generated so far) through every decoder block, though in practice this is made more efficient via KV caching (Module 16), which avoids fully recomputing attention for the already-processed prompt tokens on every single step.

Architecture

Q: What’s the relationship between the LM head and the token embedding matrix from Module 2?

Ans: Both are matrices of shape related to vocab_size × d_model — the embedding matrix maps token IDs to embeddings (a lookup), while the LM head maps a final hidden state to logits over the vocabulary (a projection).

They serve inverse-ish roles (embedding: token → vector; LM head: vector → scores over tokens), and in many real model architectures, these two matrices are even “tied” — sharing the same learned weights — as a practical parameter-efficiency technique.

Engineering

Q: Why does response latency for an LLM API call scale with the number of tokens generated, not just the prompt length?

Ans: Because, as traced in this module, generating each new token requires a full forward pass through the entire model — this repeats once per output token. A longer prompt increases the cost of each individual forward pass (more tokens to process attention over), but generating more output tokens means running that entire forward pass process more times in sequence.

Both factors contribute to total latency, which is exactly why techniques like KV caching (Module 16) that reduce the per-step cost of this repeated process matter so much for real-world serving performance.

AI Engineering

Q: If you wanted to explain to a non-technical stakeholder why an LLM sometimes “loses track” of something mentioned early in a very long conversation, how would you connect this to what you learned in this module?

Ans: I’d explain that everything the model “knows” about the conversation at any point is entirely contained in the hidden representations built up through this exact trace — nothing is separately “remembered” outside of what attention (Module 3-7) manages to incorporate into the current sequence’s representations.

If relevant information was mentioned very early in a long conversation, the model’s attention still has direct access to it (Module 6-7’s distance-independent attention), but with enough competing information in a long context, that early detail may simply receive less attention weight relative to more recent, salient context — not because it’s been “forgotten” in a human sense, but because of how attention naturally distributes its focus.

17. What You Should Remember

The complete LLM trace: prompt → tokens → embeddings + positional info → stack of causal decoder blocks → last position’s hidden state → LM head → logits → softmax → next token — verified directly, end to end, on a real (tiny) example.
Only the last position’s hidden state is used to predict the next token during inference, even though every position has its own hidden state.
“LLM generation is repeated next-token prediction” is now a mechanically traced fact, not a slogan — the entire trace repeats once per generated token.

18. How This Helps Me Build AI Systems

You’ve now traced, with real numbers, the complete mechanism behind every LLM interaction you’ll ever build on top of. Nothing about “how an LLM works” should feel like an unexplainable black box — Module 15 covers how the weights driving this exact trace get trained in the first place, and Module 16 covers how this repeated process is made efficient enough to serve in production.

Next: Module 15 — Training a Transformer — connecting this exact forward pass to the loss, backpropagation, and optimizer mechanics you already know from the Deep Learning course.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026