The Decoder and Next-Token Prediction articles both described the same repeated process — predict a token, add it to the sequence, predict again — without ever naming the formal, technical term for it. That term is autoregressive generation.
The simple definition
Autoregressive generation is the process of generating output by repeatedly feeding a model’s own previous output back in as part of its new input. The word breaks down cleanly: “auto” (self) plus “regressive” (regressing, or referring back) — a model predicting its next output using its own past outputs as input. This is precisely the mechanism the Next-Token Prediction article described in detail; this article simply gives that mechanism its proper, formal name — one you’ll see constantly in AI research papers and technical documentation.
Why this term exists separately from “next-token prediction”
It’s worth being precise about the relationship between these two closely related ideas. Next-token prediction, as covered in its own article, describes the specific task — predict what token comes next. Autoregressive generation describes the process of repeatedly performing that task, feeding each new prediction back in as input for the next round — recall the exact loop from the Next-Token Prediction article: predict, append, repeat. “Autoregressive” is a term borrowed from statistics, where it originally described predicting a future value in a time series using that same series’ own past values — and it turns out to describe language generation with striking precision, since a model predicting its next word using the words it already generated is doing exactly the same kind of self-referential prediction.
flowchart LR
A["'The'"] --> B[Model predicts next token]
B --> C["'The sky'"]
C --> D[Model predicts next token, using its own prior output as input]
D --> E["'The sky is'"]
E --> F["...continues, autoregressively"]ANALOGY vs. TECHNICAL REALITY
Analogy: Think of a meteorologist predicting tomorrow’s temperature partly by looking at today’s temperature, and predicting the day after that partly by looking at the temperature they just predicted for tomorrow — each new prediction feeding directly into the next one, building a forecast chain where later predictions depend on earlier ones, including the model’s own prior guesses.
Where this breaks down: A meteorologist’s forecast chain can be revisited and corrected with fresh weather data as it comes in, catching and correcting earlier mistakes. An autoregressive language model has no such correction mechanism once a token is generated and added to the sequence — as covered in the Next-Token Prediction article’s discussion of hallucination, once a token is chosen, it becomes a fixed, unchangeable part of the input for every subsequent prediction, with the model never revisiting or reconsidering that earlier choice.
Why this term matters for understanding a model’s real limitations
This connects directly to a genuine, structural limitation worth naming plainly. Because generation is autoregressive — each new token depends on everything generated before it, including the model’s own prior tokens — an early mistake can compound. If a model generates a subtly wrong or misleading token early in a response, every subsequent token gets generated in the context of that error being treated as established fact, since the model has no mechanism to go back and revise what it already committed to. This is a genuine, well-understood consequence of the autoregressive architecture itself, distinct from but closely related to the hallucination discussion in the Next-Token Prediction and Probability Distribution articles.
What autoregressive generation is contrasted against
It’s worth knowing the alternative this term is implicitly contrasted with, since it clarifies why the word matters at all. Recall from the Encoder article that an encoder processes an entire sequence simultaneously, non-autoregressively — all positions computed at once, with no dependency on the model’s own prior output within that same pass. Autoregressive generation is specifically the opposite pattern: sequential, self-referential, one step genuinely depending on the last. This is exactly why decoder-only models, as covered in the Decoder article, are described as autoregressive language models, while encoder-only models like BERT are not — they don’t generate a sequence of new tokens one at a time at all.
A concrete example, layered
For a simple beginner example: a small autoregressive model completing “Once upon a” generates “time,” then uses “Once upon a time” (including its own newly generated word) as the input for predicting the next token, and so on — every single step genuinely depends on the model’s own prior output, not just the original prompt. For a production example: GPT-3, GPT-4, Gemini, and Claude are all autoregressive language models — every response they generate is produced through exactly this repeated, self-referential loop, confirmed directly by the decoder-only architecture covered in the Decoder article, generating potentially thousands of tokens per response, each one genuinely dependent on everything the model has already generated in that same response.
Common misconception
A frequent beginner assumption: that “autoregressive” describes something automatic or effortless about generation, based on the word “auto” alone. As this article has explained, “autoregressive” is a precise, borrowed statistical term describing a specific dependency structure — predictions depending on the model’s own past outputs — not a casual description of ease or automation. It’s worth knowing the term specifically because it’s used constantly, without further explanation, throughout AI research papers and technical documentation describing exactly the generation process this glossary has built up across the last several articles.
Where this fits in the broader picture
You now have the complete, formal vocabulary for how modern language models actually generate text: autoregressive generation, built on next-token prediction, powered by a decoder’s masked self-attention, stabilized by residual connections and layer normalization, and made computationally practical through the KV cache. Together with the rest of this phase, this closes out the full mechanical picture of how a Transformer-based language model actually produces the text you read from it.
In one sentence
Autoregressive generation is the formal term for a model repeatedly feeding its own prior output back in as new input to predict what comes next — the precise, self-referential process underlying every response a decoder-only language model like GPT, Gemini, or Claude has ever generated, and a structure with genuine, direct consequences for why early mistakes in a response can compound.
Related Terms
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed