Encoder

The Transformer article introduced two working halves without explaining either in full. This article covers the first: the encoder — the half whose job is reading and deeply understanding input.

The simple definition

An encoder is the part of a Transformer that processes an entire input sequence at once and produces a rich, contextual internal representation of it — a set of vectors capturing not just each individual token, but how every token relates to every other token in the sequence. Recall from the Embedding article’s discussion of vectors capturing meaning — an encoder’s output is exactly this idea, applied to a whole sequence at once, with every token’s representation shaped by full awareness of every other token around it.

Why “reading the whole thing at once” matters

Recall from the Transformer article that the whole architecture’s breakthrough was processing sequences in parallel rather than strictly in order. An encoder embodies this directly: it sees the entire input sequence simultaneously — every token attending to every other token, both before and after it — building a representation where each token’s meaning is fully informed by its complete surrounding context, in both directions. This is a genuinely different capability from only ever looking backward at what came before, which the Decoder article, covered next, will show is exactly how a decoder works instead.

flowchart LR
    A["Full input: 'The bank by the river'"] --> B[Encoder: every token sees every other token, both directions]
    B --> C["Rich contextual representation: 'bank' correctly understood as riverbank, not financial bank, because 'river' is visible too"]

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of a translator reading an entire foreign-language paragraph completely before starting to translate a single word of it — absorbing the full context first, so that an ambiguous word early in the paragraph can be correctly understood in light of clarifying details that appear later on. Only after this full, complete reading does the actual translation work begin.

Where this breaks down: A human translator’s “full reading” involves genuine comprehension and memory. An encoder’s “full reading” is the mechanical, parallel attention calculation covered in the Attention and Self-Attention articles, applied simultaneously across the whole input — every token’s representation gets updated based on every other token’s, all at once, through matrix calculations, not sequential human-style reading.

Why this bidirectional view is genuinely powerful

This connects directly to a real, well-documented example worth knowing: the word “bank” in “the bank by the river” versus “the bank approved my loan” means something completely different depending on nearby words — and in the first sentence, the disambiguating word “river” comes after “bank.” An encoder, seeing the whole sentence at once, can use that later context to correctly interpret the earlier, ambiguous word — something a model that could only look backward, one direction at a time, would genuinely struggle with.

Where encoder-based models actually get used

Encoders excel specifically at understanding tasks — tasks where the goal is to deeply comprehend an input and produce some judgment or representation about it, rather than to generate new, extended text. BERT (Bidirectional Encoder Representations from Transformers), a landmark model published by Google researchers, is a well-known, real example of an encoder-only architecture — built specifically for tasks like classification, sentiment analysis, and the kind of semantic understanding that powers the Embedding Models covered in the Data Representation phase. Recall from that phase that embedding models need to deeply understand a whole piece of text at once to produce a good embedding — exactly the kind of task an encoder architecture is naturally suited for.

A concrete example, layered

For a simple beginner example: an encoder processing the sentence “I love this movie” for sentiment classification sees the entire sentence at once, and its final representation captures the overall positive sentiment, informed by every word’s relationship to every other word, ready to be classified as “positive” by a simple output layer on top. For a production example: many production embedding models, including the general architecture underlying models like OpenAI’s text-embedding-3-small covered in the Embedding Model article, are built on encoder-style Transformer architectures specifically because their entire job — producing one rich, meaning-capturing vector for a whole piece of input text — benefits directly from the encoder’s full, bidirectional view of the input.

Follow a sentence through an encoder

The animal did not cross the road because it was tired.

Every token can attend to tokens on both sides. The representation for it can use animal, road, and the surrounding grammar to build contextual meaning.

flowchart LR
    A[All input token vectors] --> B[Bidirectional self-attention]
    B --> C[Contextual vector for every token]
    C --> D[Classification, retrieval, or decoder cross-attention]

How encoders relate to GPT and Gemini

Classic GPT language models are decoder-only; they do not first pass text through a separate bidirectional text encoder. That fits next-token generation.

Encoder models remain useful for embeddings, classification, search, and understanding tasks. Multimodal Gemini systems also convert images, audio, and video into representations the model can process. The Gemini 1.0 report describes visual encoding inspired by Flamingo, CoCa, and PaLI.

Encoder-only model: input → contextual representations → classification or embedding
Decoder-only GPT:    previous tokens → causal representations → next token
Multimodal Gemini:   media encoding + text representations → Transformer processing

Three practical encoder examples

Example 1: classify a support message

For “My payment was taken twice,” every token can look both left and right. The encoder produces context-aware vectors, and a small classification head can map the final representation to billing_problem.

Example 2: create an embedding for search

An embedding model can encode “How can I reset my password?” into one dense vector. A search system compares it with encoded help articles and can retrieve “Recover access to your account” even though the wording differs.

Example 3: understand an image before answering

In a multimodal system, a visual encoder can turn image regions into numeric representations. Gemini’s architecture report explains that visual inputs are encoded and interleaved with other modalities, allowing later Transformer processing to connect a chart, a written question, and relevant labels.

raw input → encoder representations → task-specific component → result
email     → contextual token vectors → classifier              → spam / not spam
sentence  → one dense vector         → vector search           → relevant document
image     → visual representations   → language decoder        → written answer

An encoder normally represents or understands supplied information. It does not automatically generate a long answer one token at a time; that is the decoder’s natural role.

Common misconception

A frequent beginner assumption: that every modern language model, including ChatGPT-style chatbots, uses this same encoder architecture. As the next article on decoders will make clear, this isn’t the case — GPT, Gemini, and Claude are built primarily around the decoder half of the Transformer, not the encoder, precisely because their core task — generating new text one token at a time — has different requirements from the “deeply understand a fixed input” task encoders excel at.

Where this fits in what comes next

You now understand the half of the Transformer built for deep, bidirectional understanding. The next article, Decoder, covers the other half — the one responsible for actually generating new text, one token at a time, and the architecture underlying the large language models covered throughout the previous phase of this glossary.

In one sentence

An encoder processes an entire input sequence at once, building a rich representation where every token’s meaning is informed by full, bidirectional awareness of every other token — the architecture behind understanding-focused tasks like classification and embeddings, rather than open-ended text generation.

Related Terms

• Transformer
• Decoder
• Attention
• Embedding Model
• Self-Attention