Next-Token Prediction

Every article in this phase has gestured toward this mechanism without fully unpacking it — “assembled piece by piece,” “token by token.” It’s time to explain the actual, literal task a modern language model performs at every single step: next-token prediction.

The simple definition

Next-token prediction is the task of guessing which token is most likely to come next, given everything that came before it. That’s genuinely the entire training objective for most modern large language models — recall from the Loss Function article’s discussion of cross-entropy loss on next-token prediction, and from the Token and Sequence articles that text gets broken into an ordered run of tokens. A language model’s whole job, at its mathematical core, is answering one repeated question: given this sequence so far, what token comes next?

Why one narrow-sounding task produces such broad ability

This directly extends the point made in the Language Model article about why next-word prediction turns out to require broad competence. To predict the next token accurately across billions of diverse sentences, a model has to implicitly learn grammar (to predict grammatically sensible continuations), facts (to predict “Paris” after “The capital of France is”), reasoning patterns (to predict the next step in a worked math problem), and style (to predict a continuation that matches the tone already established). None of these were taught as separate lessons — they all emerge as side effects of getting good at the single, repeated task of next-token prediction, applied across an enormous amount of real text.

flowchart LR
    A["Sequence so far: 'The sky is'"] --> B[Model calculates probability for every possible next token]
    B --> C["'blue': 45%"]
    B --> D["'clear': 20%"]
    B --> E["'falling': 2%"]
    B --> F["...thousands more possibilities"]

How a full response actually gets built, one token at a time

This is worth walking through concretely, since it explains something that can otherwise feel mysterious about how a chatbot produces a whole paragraph. Recall from the Inference article’s discussion of the decode stage: the model doesn’t generate an entire response in one shot. It predicts one single next token, adds that token to the sequence, then repeats the entire process again — now predicting the next token after that, based on the sequence including the token it just added — over and over, until it produces a special token signaling the response is complete. A fluent, multi-paragraph answer from a chatbot is really hundreds or thousands of individual next-token predictions, chained together, each one building on everything generated so far.

flowchart LR
    A["'The'"] --> B["Predict next: 'sky'"]
    B --> C["'The sky'"] --> D["Predict next: 'is'"]
    D --> E["'The sky is'"] --> F["Predict next: 'blue'"]
    F --> G["...continues, one token at a time"]

Training can score many positions; generation remains sequential

During training, one sequence provides many learning targets:

Input: [The]          Target: sky
Input: [The, sky]     Target: is
Input: [The, sky, is] Target: blue

With a causal mask, Transformer training can calculate losses for these positions in parallel. During generation, the next token does not exist yet, so the model must select it, append it, and then run the next step.

Training:  many known target positions can be scored together
Inference: token 1 → token 2 → token 3, sequentially

How generation stops

The loop ends when:

• The model produces an end-of-sequence or configured stop token.
• A configured stop sequence appears.
• The maximum output-token limit is reached.
• The application or user cancels generation.

Stopping is part of the decoding system; it does not prove the model decided its answer was complete in a human sense.

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of a skilled improv performer building a scene one line at a time, each line responding to everything said so far, without knowing in advance exactly where the scene will end up — the whole performance emerges from a long chain of individual, in-the-moment responses, each one shaped by the accumulated context of everything that came before it.

Where this breaks down: An improv performer has genuine creative intent and a sense of narrative direction. Next-token prediction has neither — each individual prediction is a calculated probability, produced by the mechanics covered in the very next two articles of this phase, with no forward-looking plan for where the “story” is heading; the appearance of a coherent, planned response emerges entirely from the accumulation of many individually well-chosen next tokens, not from any actual planning happening in advance.

A concrete example, layered

For a simple beginner example: given the sequence “Once upon a”, a small language model trained on children’s stories would very likely predict “time” as the single most probable next token, based on how overwhelmingly common that exact phrase is in its training data. For a production example: GPT-3, trained via exactly this next-token prediction objective across roughly 300 billion tokens as confirmed in OpenAI’s published paper, generates every single response you’d get from it by repeating this one mechanical step — predict the next token, append it, repeat — potentially thousands of times in a row to produce one long, coherent answer.

Why this simple mechanism has real, honest limitations

It’s worth being direct about a genuine consequence of this architecture, one that connects back to a caution raised repeatedly throughout this glossary. Because a model predicts only the single next token at each step, based purely on learned statistical patterns rather than verified fact-checking or forward planning, it has no built-in mechanism to “look ahead” and confirm a multi-step answer is actually correct before committing to it — this is a genuine structural contributor to the hallucination problem discussed in the Generative AI article, not merely a training data gap that more data alone would fully fix.

Follow one real generation loop

Prompt:

The sky is

At one step, imagine the model assigns these illustrative probabilities:

blue  = 0.60
clear = 0.20
dark  = 0.10
wide  = 0.06
...all other tokens together = 0.04

If blue is selected, the sequence becomes The sky is blue. The entire model runs again to predict the next token—perhaps punctuation, today, or another continuation.

flowchart LR
    A[The sky is] --> B[Predict distribution]
    B --> C[Select blue]
    C --> D[The sky is blue]
    D --> E[Predict again]

The displayed words are illustrative. A real GPT or Gemini model scores its full tokenizer vocabulary, including word pieces, spaces, punctuation, and special tokens.

Real GPT and Gemini evidence

OpenAI describes GPT-3 as an autoregressive language model trained at 175B-parameter scale. OpenAI’s instruction-following article also directly describes the base training goal as predicting the next word on large Internet text before post-training improves instruction following.

The Gemini 1.5 report discusses next-token prediction across very long contexts. Gemini extends the available context with multimodal representations, but text generation still proceeds through token predictions.

Common misconception

A frequent beginner assumption: that a language model somehow plans out an entire response in advance, then writes it down. As the “how a full response gets built” section explained, this isn’t how the underlying mechanism actually works — there’s no advance outline or plan; a coherent, well-structured response emerges purely from a long chain of individually good next-token predictions, each one only “seeing” the sequence so far, with genuinely no advance knowledge of what comes several tokens later.

Where this fits in what comes next

You now understand the actual mechanical task every prediction step performs. The next article, Logits, covers exactly what the model calculates internally before it settles on a probability for each possible next token — the raw, unprocessed numbers this whole prediction process starts from.

In one sentence

Next-token prediction is the single, repeated task every modern language model actually performs — guess the next token, add it to the sequence, repeat — and it’s the surprisingly simple mechanical engine underneath every fluent response a chatbot has ever produced.

Related Terms

• Token
• Sequence
• Prediction
• Loss Function
• Logits