The Self-Attention article closed on a genuine, real gap: self-attention compares every token against every other token simultaneously, with no inherent sense of order. Multi-Head Attention and the Feed-Forward Network, Residual Connections, and Layer Normalization that surround them fill out the rest of a Transformer layer’s internal machinery, but none of them solve that ordering gap either. This article covers the fix: positional encoding.
The simple definition
Positional encoding is a technique for injecting information about each token’s position in a sequence directly into its numeric representation, so that a Transformer — which otherwise processes tokens in parallel with no built-in sense of order — can still tell the difference between different word orders. Recall from the Sequence article that order carries essential meaning — “the dog bit the man” versus “the man bit the dog.” Without positional encoding, a Transformer’s self-attention, comparing every token against every other simultaneously, would treat both of these sentences as containing exactly the same information, since it would have no way to know which token came first.
Why this problem exists at all, and why it’s a genuinely necessary fix
Recall from the Transformer article’s core insight: parallel processing, all tokens at once, is precisely what made the Transformer so much faster than older, strictly sequential architectures. But that same parallelism is exactly what creates this problem — an older, sequential architecture naturally knew order, simply because it processed tokens one after another, in order, by construction. A Transformer, processing everything at once, gets speed but loses that automatic sense of order as a direct trade-off — and positional encoding is the specific, deliberate patch that restores what parallel processing gave up.
flowchart LR
A[Tokens processed in parallel] --> B[No inherent sense of order]
B --> C[Positional Encoding: injects position information directly]
C --> D[Self-attention can now distinguish word order]How positional encoding actually works
The original 2017 Transformer paper’s solution, worth understanding concretely, adds a specific, mathematically generated pattern of numbers to each token’s embedding — recall from the Embedding article that a token’s embedding is already a vector capturing meaning. Positional encoding adds a second vector, encoding the token’s position (1st, 2nd, 3rd, and so on) using a specific combination of sine and cosine wave patterns, directly onto that meaning-vector, producing one combined vector that carries both “what this token means” and “where in the sequence it sits.” This particular wave-based approach was chosen specifically because it lets the model calculate relative distances between positions consistently, and can, in principle, generalize to sequence lengths longer than what the model was directly trained on.
flowchart LR
A[Token's meaning: Embedding vector] --> C[Add together]
B[Token's position: Positional Encoding vector] --> C
C --> D[Combined vector: meaning + position, fed into self-attention]ANALOGY vs. TECHNICAL REALITY
Analogy: Think of a stack of numbered raffle tickets, shuffled into a pile. Even after shuffling, each ticket still has its own printed number on it, so you can always reconstruct the original order they were issued in, regardless of how mixed up the physical pile becomes. Positional encoding works the same way — even though self-attention processes every token “all mixed together” in parallel, each token still carries its own built-in “number,” letting the model reconstruct and use order information despite the parallel processing.
Where this breaks down: A raffle ticket’s printed number is a simple, literal integer. Positional encoding’s actual numbers are a more elaborate mathematical pattern (the sine and cosine combination described above), chosen specifically because it has useful mathematical properties for the self-attention calculations covered in the previous two articles — not a simple counting number, but a carefully engineered signal designed to work well with the rest of the architecture.
A concrete example, layered
For a simple beginner example: processing “dog bites man” versus “man bites dog,” positional encoding ensures “dog” carries a different combined vector depending on whether it appears in position 1 or position 3 — even though the word “dog” itself, and its base meaning-embedding, is identical in both sentences, its position-adjusted representation is genuinely different, letting the model correctly distinguish the two sentences’ very different meanings. For a production example: GPT-3’s published architecture, as confirmed in OpenAI’s paper, uses positional information as a core part of how it processes its full context — every one of its 96 layers works with token representations that carry position information, letting the model correctly track word order across sequences up to its full context window, a concept covered in full in the next article.
Why this remains an active area of ongoing refinement
It’s worth noting honestly that the original 2017 sine-and-cosine approach isn’t the only technique in use today — it’s one of several real, published approaches, and different modern models make different choices here. Some newer architectures use alternative positional encoding schemes specifically designed to generalize better to longer sequences than a model was originally trained on, an active, ongoing area of research rather than a single, permanently settled solution — a genuine limitation of the original approach that continues to motivate real engineering work across the field.
Add token and position information
Suppose the token embedding is [0.2, 0.7] and its position vector is [0.1, -0.2]:
token embedding [0.2, 0.7]
position vector + [0.1, -0.2]
-----------
model input [0.3, 0.5]The same word at a different position receives different positional information, allowing attention to distinguish order.
Common position strategies
| Strategy | Core idea |
|---|---|
| Sinusoidal encoding | Add fixed sine/cosine patterns, used in the original Transformer. |
| Learned absolute positions | Learn a vector for position 0, 1, 2, and so on. |
| Relative position methods | Represent how far tokens are from one another. |
| Rotary position embeddings | Rotate query/key components based on position. |
How GPT and Gemini use positional information
OpenAI’s public GPT-2 implementation contains a learned position-embedding table in addition to token embeddings. Later model families can use different methods.
Gemini’s long-context capabilities require positional mechanisms that remain useful across long sequences, but high-level Gemini reports do not expose every low-level implementation detail. The safe mental model is consistent: token identity alone is insufficient; the Transformer must also encode order or relative position.
flowchart LR
A[Token identity] --> C[Position-aware representation]
B[Order or relative position] --> C
C --> D[Attention]What would fail without position information?
Imagine these two inputs contain the same three token embeddings:
dog bites man
man bites dogWithout any position signal, self-attention receives the same set of token meanings in both cases. The calculations can rearrange with the tokens but have no independent marker saying which one was first, second, or third. Position information breaks that symmetry.
Absolute position and relative distance
Two questions are useful:
- Absolute: “Is this token at position 42?”
- Relative: “Is this token three places before that token?”
Learned absolute embeddings are similar to assigning every seat a learned ID. Relative and rotary approaches help attention represent distances and order relationships between tokens. This matters in long sequences because the model must distinguish “the previous line” from “a line thousands of tokens earlier.”
A tiny position-dependent example
Base embedding for “dog” [0.2, 0.7]
Position information when dog is first +[0.1,-0.2] → [0.3,0.5]
Position information when dog is third +[0.4, 0.1] → [0.6,0.8]The displayed vectors are deliberately tiny teaching numbers. Production models use hundreds or thousands of dimensions and may combine position with attention through a method such as rotary position embeddings instead of literally using this exact addition.
Common misconception
A frequent beginner assumption: that a Transformer somehow “just knows” word order automatically, the same intuitive way a person reading a sentence does. As this article has explained, this isn’t automatic at all — it requires this specific, deliberately engineered fix, precisely because the underlying self-attention mechanism, covered in the previous two articles, has no built-in sense of order on its own. Positional encoding isn’t a minor implementation detail; it’s a genuinely necessary component, without which a Transformer literally could not distinguish “dog bites man” from “man bites dog.”
Where this fits in what comes next
You now understand how a Transformer restores the sense of order its parallel processing would otherwise lose. The next two articles, Context Window and Context Length, cover a directly related, practical question: exactly how much sequence — how many tokens, in what positions — a model can actually keep track of at once.
In one sentence
Positional encoding injects information about each token’s position directly into its representation, restoring the sense of word order that a Transformer’s parallel, all-at-once processing would otherwise lose entirely — a deliberately engineered fix, not an automatic byproduct of the architecture.
Related Terms
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed