Transformer

The journey through this phase

flowchart LR
    A[Tokens and embeddings] --> B[Positional information]
    B --> C[Attention and self-attention]
    C --> D[Encoder, decoder, or decoder-only blocks]
    D --> E[Context window limits what fits]
    E --> F[KV cache speeds generation]

This phase opens the architecture behind GPT and Gemini-style models and separates the original Transformer from the decoder-only form commonly used for generation.

Every model name mentioned throughout this glossary — GPT, Gemini, Claude, Llama — shares one underlying architecture, referenced repeatedly but never fully explained. It’s time to open it up: the Transformer.

The simple definition

The Transformer is a neural network architecture, introduced in 2017, built entirely around a mechanism called attention — letting every part of a sequence directly consider every other part, all at once, rather than processing information strictly in order, one step at a time. Recall from the Algorithm article’s early mention of the Transformer as the architecture behind GPT, Gemini, and Claude — this article finally explains what that architecture actually is and why it replaced everything that came before it.

What came before, and why it had a real, limiting problem

Before the Transformer, the dominant approach to processing sequences — like the token sequences covered in the Sequence article — used architectures that worked through a sequence strictly one element at a time, in order, carrying forward a summary of everything seen so far. This sequential requirement created a genuine bottleneck: each step had to wait for the previous one to finish, making these older architectures slow to train (since GPUs, as covered in the Training article, excel at doing many calculations simultaneously, not one at a time) and prone to “forgetting” information from far earlier in a long sequence by the time they reached the end of it.

The core insight: process the whole sequence at once, using attention

The Transformer’s breakthrough insight, published by Vaswani and colleagues at Google in the paper “Attention Is All You Need,” was to abandon strict, sequential processing entirely. Instead, every token in a sequence can directly look at — “attend to” — every other token simultaneously, regardless of how far apart they are, through the Attention mechanism covered fully in the next two articles of this phase. This single change unlocked massive parallelization (all tokens processed together, ideally suited to GPU hardware) and dramatically better handling of long-range relationships between distant parts of a sequence.

flowchart LR
    A[Old sequential approach: process one token at a time, in strict order] --> B[Slow, struggles with long-range dependencies]
    C[Transformer: all tokens processed together, using attention] --> D[Fast, parallelizable, strong long-range understanding]

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of the difference between reading a mystery novel one word at a time, trying to remember every clue purely from memory as you go, versus having the entire book laid out in front of you, able to instantly flip back and directly compare any two passages, no matter how far apart they are, whenever a new detail makes an earlier clue suddenly relevant.

Where this breaks down: A person flipping through a book uses conscious judgment about which passages to compare. The Transformer’s attention mechanism does this “flipping back” mathematically and mechanically, for every single token against every other token, calculated through the specific mechanism the next two articles will explain in full — not a deliberate, judgment-based search, but a calculated weighting applied uniformly across the whole sequence.

The two halves: encoder and decoder

The original Transformer architecture, as published in 2017, consists of two main components working together: an encoder, which processes an input sequence and builds a rich internal representation of it, and a decoder, which generates an output sequence, using both what it’s generated so far and the encoder’s representation of the input. These two components get their own full treatment in the next two articles of this phase — and, notably, many modern large language models, including GPT, actually use only one of these two halves, a distinction those articles will make precise.

flowchart LR
    A[Input sequence] --> B[Encoder: builds rich internal representation]
    B --> C[Decoder: generates output sequence]
    C --> D[Output sequence]

A concrete example, layered

For a simple beginner example: translating “The cat sat on the mat” from English to French requires understanding that “sat” relates back to “cat” (who is doing the sitting) even though other words sit between them — exactly the kind of long-range relationship the Transformer’s attention mechanism was specifically designed to capture directly, rather than relying on a fading, sequential memory of earlier words. For a production example: GPT-3, GPT-4, Gemini, Claude, and Llama are all, at their architectural core, Transformers — specifically, as covered in the next article, decoder-only variants of the original 2017 design, scaled up to the hundreds of billions of parameters covered throughout the Parameters and Large Language Model articles.

Why the Transformer’s arrival mattered so much

It’s worth being direct about how significant this shift actually was, not just presenting it as one architectural choice among many. The original 2017 paper itself demonstrated the Transformer achieving better translation quality than the prior best models, while being significantly faster to train — a genuine, measured improvement, not just a theoretical one. Every major advance in language AI covered throughout this glossary — from GPT-3’s emergent capabilities discussed in the Large Language Model article, to the RAG systems covered in the Vector Database article — builds directly on this one 2017 architectural foundation.

What is inside one Transformer block?

flowchart TB
    A[Input representations] --> B[Self-attention]
    B --> C[Residual connection and normalization]
    C --> D[Feed-forward network]
    D --> E[Residual connection and normalization]
    E --> F[Output representations]

Attention is only one component. Feed-forward layers transform each token representation, residual connections preserve information paths, and normalization supports stable training.

Real architecture data: original Transformer, GPT-3, and Gemini

The original Attention Is All You Need base Transformer had 6 encoder layers, 6 decoder layers, hidden size 512, and 8 attention heads.

OpenAI’s published GPT-3 175B configuration used a decoder-only Transformer with 96 layers, hidden size 12,288, 96 attention heads, and context length 2,048 tokens.

The Gemini 1.0 report says Gemini builds on enhanced Transformer decoders and supports interleaved text with images, audio, and video.

Architecture	Structure	Typical strength
Original Transformer	Encoder and decoder	Sequence-to-sequence translation
GPT-3	Decoder-only	Autoregressive text generation
Gemini	Multimodal encoding with Transformer decoder foundation	Multimodal understanding and generation

Follow one request through a modern Transformer

Suppose you ask, “Why do leaves look green?” A GPT- or Gemini-style system does not send the sentence through one magical box. It performs a chain of smaller operations:

flowchart TD
    A[Text or multimodal input] --> B[Convert input into tokens]
    B --> C[Turn tokens into embeddings]
    C --> D[Add position information]
    D --> E[Run Transformer block 1]
    E --> F[Run many more Transformer blocks]
    F --> G[Produce logits for possible next tokens]
    G --> H[Choose one token, such as Because]
    H --> I[Repeat until the answer is complete]

Inside each block, attention lets a token collect relevant information from other tokens, while the feed-forward network transforms that token’s updated representation. Residual connections preserve earlier information and normalization keeps the calculations stable. The output of one block becomes the input of the next, so the representation becomes more useful layer by layer.

The exact meaning of an individual layer is not hand-written by an engineer. During training, the model learns distributed patterns across many layers and nodes. One layer may strengthen word relationships while later layers combine grammar, facts, instructions, and task-specific clues—but these jobs overlap rather than forming perfectly named departments.

Common misconception

A frequent beginner assumption: that “Transformer” refers to some specific product or company’s proprietary technology, similar to how “GPT” refers to OpenAI’s specific model family. As this article has explained, the Transformer is an openly published, general architecture — introduced by Google researchers and freely available for any organization to build upon — which is exactly why so many different companies (OpenAI, Google, Anthropic, Meta) have each built their own distinct models using the same underlying Transformer foundation.

Where this fits in what comes next

You now understand the Transformer’s core insight — parallel, attention-based processing replacing strict sequential order. The next two articles, Encoder and Decoder, cover the two structural halves introduced in this article in full detail, followed by Attention and Self-Attention, which explain exactly how the “attend to every other token” mechanism this article described actually works, mathematically.

In one sentence

The Transformer is the 2017 architecture that replaced strict, sequential processing with parallel, attention-based processing — letting every token directly consider every other token at once — and it’s the single architectural foundation underneath essentially every major language model covered throughout this glossary.

Related Terms

• Attention
• Encoder
• Decoder
• Sequence
• Large Language Model (LLM)