Positional Information | TechByteByByte

Begin with the central question

If attention can compare every token with every other token, how does it know which token came first?

Essential words

Self-attention alone is permutation-equivariant: reordering inputs reorders outputs without inherently encoding order. A positional encoding or position embedding adds order information. RoPE rotates vector components according to position.

What You Will Understand

A direct, numerically verified proof that self-attention alone cannot tell “dog bites man” from “man bites dog” — and how positional information fixes this. You’ll cover sinusoidal encoding, learned positional embeddings, and RoPE, with intuition weighted more heavily than derivation, per this course’s scope.

token embedding + position signal -> order-aware representation

The problem this module solves

Module 1 proved attention’s computations are order-independent — a genuine advantage for parallelization. But that same property is a serious problem for language, where order carries meaning. Positional information exists to reintroduce exactly the order-sensitivity that attention’s parallelizable design otherwise discards.

Build the intuition

attention asks “how relevant is token A to token B,” based purely on their content — it has no built-in sense of “A comes before B” or “A comes three positions after B.” Positional information is a way of stamping each token’s embedding with where it sits, so that content and position both influence the final relevance computation.

4. Real-World Analogy

Imagine handing someone a shuffled deck of index cards, each with one word from a sentence, no page numbers. They can tell you what words are present and even guess likely relationships, but they can’t reconstruct the original sentence’s meaning without knowing the order the cards were dealt in. Positional encoding is the page number written on each card.

Analogy: The Shuffled Index Cards (Absolute Page Stamp vs. Relative Angle Dial) Imagine trying to reconstruct a story from a pile of index cards:

The Problem: The cards are dealt face-down in random locations. If you only look at word text, “dog bites man” and “man bites dog” use the exact same cards. You need sequence structure.

Absolute Stamps (Sinusoidal / Learned): You take a blue marker and write the exact physical card index on the card’s margin: “Position 0”, “Position 1”, “Position 2”. (This is additive absolute positional encoding).

Rotary Rotation (RoPE): Instead of writing numbers, you attach a small compass dial to each card.

You rotate the arrow of the dial by 30 degrees for card 1, 60 degrees for card 2, and 90 degrees for card 3. When comparing card 1 (30°) to card 3 (90°), the absolute degrees don’t matter as much as the relative angle difference (60° separation).

This relative rotation helps LLMs read documents much longer than those they were trained on, because the relative separation angle generalizes cleanly.

📊 Visual Chart: Absolute vs. Rotary Positional Encodings

Here is how absolute additions differ from relative rotations inside the coordinate space:

graph TD
    subgraph Absolute ["Absolute Addition (Sinusoidal/Learned)"]
        TokenEmb["Base Token Vector: X"] --> AddGate["Vector Addition: +"]
        PosVec["Positional Vector: PE"] --> AddGate
        AddGate --> TransInput["Final Position-Aware Input Vector"]
    end




    subgraph Rotary ["Relative Rotation (RoPE)"]
        RawQ["Raw Query Vector: Q"] --> RotateQ["Rotate 2D segments by angle (m x theta)"]
        RawK["Raw Key Vector: K"] --> RotateK["Rotate 2D segments by angle (n x theta)"]




        RotateQ --> DotProd["Relative Distance Dot Product:<br>(Rotated Q) . (Rotated K)"]
        RotateK --> DotProd
    end

5. Core Concept

"Dog bites man"  vs.  "Man bites dog"




Same three words. Same set of pairwise token relationships,
AS FAR AS CONTENT-ONLY ATTENTION CAN TELL. Completely
different meaning.

Term	Definition
Positional encoding (sinusoidal)	A fixed, non-learned function generating a unique vector per position, using sine/cosine at varying frequencies
Learned positional embeddings	A trainable embedding table (like token embeddings), one row per position, learned via gradient descent
RoPE (Rotary Positional Embedding)	A modern technique that encodes relative position by rotating Q and K vectors, rather than adding a separate positional vector

6. How It Works — Step by Step

The core problem, precisely:

1. Self-attention computes relevance between tokens using ONLY
   their content vectors (Module 3-5)
2. Two sentences with the SAME words in DIFFERENT order produce
   IDENTICAL token embeddings (Module 2 -- embedding lookup only
   depends on token ID)
3. Without positional information, attention has NO way to
   distinguish these two sentences' word arrangements

Sinusoidal encoding:

1. For each position, compute a unique vector using sine and
   cosine functions at different frequencies across the
   embedding dimensions (exact formula in DL Module 16 --
   reused here, not re-derived)
2. ADD this vector to the token embedding at that position
3. Now the SAME token at different positions produces
   DIFFERENT final vectors entering attention

Learned positional embeddings:

1. Create a trainable embedding table, one row per possible
   position (e.g., position 0 through max_sequence_length)
2. Look up and ADD the row matching each token's position --
   structurally identical to token embedding lookup (Module 2),
   just indexed by position instead of token ID
3. These positional vectors are LEARNED via gradient descent,
   rather than fixed by a sine/cosine formula

RoPE, conceptually:

1. Instead of ADDING a separate positional vector, RoPE ROTATES
   the Query and Key vectors by an angle that depends on their
   position
2. This means the DOT PRODUCT between a rotated Query and a
   rotated Key naturally encodes their RELATIVE distance --
   not just their absolute positions
3. This relative-position property is why RoPE has become
   popular for modern LLMs -- it tends to generalize better to
   sequence lengths not seen during training

7. Mathematical Intuition

Read the mathematics as a story

Without a position signal, attention knows token content but not inherent order. Adding or rotating by position makes equal token embeddings distinguishable by location.

token embedding + position 0 -> first occurrence
same embedding + position 4 -> later occurrence

RoPE’s key mathematical property, stated at the intuition level this course targets: rotating two vectors by angles proportional to their positions, then taking their dot product, produces a result that depends mathematically on the difference between their positions, not their absolute positions individually.

This is genuinely different from sinusoidal/learned encoding, where position information is added to each vector independently, with no automatic relative-distance property built into the dot product itself.

8. Small Worked Example

Walk through the example

Create sinusoidal vectors for several positions. 2. Add them to repeated token embeddings. 3. Confirm position changes the representation. 4. Observe how patterns vary smoothly.

The proof structure: embed “dog,” “bites,” “man” and “man,” “bites,” “dog” — same three token embeddings, different order. Run identical self-attention weights on both. If the two sentences’ “bites” representations come out identical, attention alone has failed to distinguish them. Add positional encoding and repeat — if the representations now differ, positional information has done its job.

9. Python / NumPy Example

What the code will demonstrate

This small NumPy example makes Positional Information visible with inspectable numbers and shapes. Read it in three passes: identify each input, follow the transformation line by line, and connect the printed output to the diagram above. The arrays are intentionally tiny teaching values; unless the text explicitly says otherwise, they are not weights or measurements from GPT, Gemini, or another trained model.

# The arrays are intentionally small so each transformation can be inspected.
# Printed values illustrate the mechanism; they are not trained-model measurements.
import numpy as np




def softmax(x, axis=-1):
    exp_x = np.exp(x - np.max(x, axis=axis, keepdims=True))
    return exp_x / np.sum(exp_x, axis=axis, keepdims=True)




np.random.seed(1)




vocab = {"dog": 0, "bites": 1, "man": 2}
d_model = 4
embedding_table = np.round(np.random.randn(3, d_model) * 0.5, 3)




def embed(words):
    return np.array([embedding_table[vocab[w]] for w in words])




sentence_a = embed(["dog", "bites", "man"])
sentence_b = embed(["man", "bites", "dog"])




Wq = np.round(np.random.randn(d_model, d_model) * 0.4, 2)
Wk = np.round(np.random.randn(d_model, d_model) * 0.4, 2)
Wv = np.round(np.random.randn(d_model, d_model) * 0.4, 2)




def self_attention(X):
    Q, K, V = X @ Wq, X @ Wk, X @ Wv
    scores = Q @ K.T / np.sqrt(d_model)
    weights = softmax(scores, axis=-1)
    return weights @ V




# --- WITHOUT positional information ---
out_a = self_attention(sentence_a)
out_b = self_attention(sentence_b)




print("'bites' representation in 'dog bites man':", np.round(out_a[1], 4))
print("'bites' representation in 'man bites dog':", np.round(out_b[1], 4))
print("Identical without positional info?", np.allclose(out_a[1], out_b[1]))




# --- WITH sinusoidal positional information ---
def positional_encoding(seq_len, d_model):
    pos = np.arange(seq_len)[:, np.newaxis]
    i = np.arange(d_model)[np.newaxis, :]
    angle_rates = 1 / np.power(10000, (2 * (i // 2)) / np.float32(d_model))
    angles = pos * angle_rates
    pe = np.zeros((seq_len, d_model))
    pe[:, 0::2] = np.sin(angles[:, 0::2])
    pe[:, 1::2] = np.cos(angles[:, 1::2])
    return pe




pe = positional_encoding(3, d_model)
sentence_a_pos = sentence_a + pe
sentence_b_pos = sentence_b + pe




out_a_pos = self_attention(sentence_a_pos)
out_b_pos = self_attention(sentence_b_pos)




print("\nWith positional info:")
print("'bites' representation in 'dog bites man':", np.round(out_a_pos[1], 4))
print("'bites' representation in 'man bites dog':", np.round(out_b_pos[1], 4))
print("Identical with positional info?", np.allclose(out_a_pos[1], out_b_pos[1]))

Expected Output:

'bites' representation in 'dog bites man': [-0.4869 -0.5652 -0.1476  0.3317]
'bites' representation in 'man bites dog': [-0.4869 -0.5652 -0.1476  0.3317]
Identical without positional info? True




With positional info:
'bites' representation in 'dog bites man': [-0.144  -0.3027  0.0425  1.1784]
'bites' representation in 'man bites dog': [-0.1054 -0.2159  0.0642  1.0724]
Identical with positional info? False

10. How It Works

Without positional information, “bites“‘s self-attention output is byte-for-byte identical (True) in both sentences — even though “dog bites man” and “man bites dog” mean opposite things, the model literally cannot tell them apart using content-only attention. This is not a subtle theoretical concern — it’s a complete failure to distinguish opposite-meaning sentences.
With positional information added, “bites“‘s representation genuinely differs (False for allclose) between the two sentences — the same word, at the same relative position (middle), but surrounded by different neighbors in different orders, now produces different self-attention output. Positional information is what makes this distinction possible at all.

11. How Is This Used in Modern AI?

Where this concept lives

Follow the concept at three levels: inside the model, where the computation happens; inside the AI product, where that computation supports a visible feature; and inside production, where engineers measure speed, memory, quality, and failure cases. The details below connect those levels.

🤖 How Is This Used in Modern AI?

Every Transformer-based model, without exception, needs some form of positional information — the specific method varies (sinusoidal, learned, or RoPE), but omitting it entirely reproduces exactly the “dog bites man” failure demonstrated above, regardless of model size.

Method	Where used
Sinusoidal	The original 2017 Transformer paper; still used in some models
Learned positional embeddings	Many earlier large models (e.g., original GPT-2/GPT-3 style)
RoPE	Most modern LLMs (e.g., LLaMA-family and many others) — the current common default

12. How Is This Used in LLMs?

Trace one model call

User text → tokens → Transformer computation → output-token probabilities
          this topic affects one part of that computation

An LLM does not apply this idea as a separate magic step. It uses it as part of the repeated numerical pipeline that transforms token vectors and produces the next-token probabilities.

RoPE’s relative-position property has a genuinely practical LLM consequence: models trained with RoPE tend to generalize somewhat better to sequence lengths longer than those seen during training, compared to fixed sinusoidal or learned absolute-position schemes — a real factor in how modern LLMs support long context windows (Module 17 covers context windows and their cost in full).

Real systems you can recognize

GPT-style and Gemini models require positional information, though production models may use learned positions, sinusoidal encodings, RoPE, or variants. Exact choices should be taken from each model’s published documentation rather than assumed.

13. How Is This Used in Agentic AI?

Trace one agent step

Goal + history + tool results
              ↓
     LLM processes the context
              ↓
Suggested answer or tool call
              ↓
Agent runtime validates and executes it

This distinction matters: the Transformer helps produce the proposal, while the surrounding agent software controls tools, permissions, retries, memory, and execution.

Direct relevance to Agentic AI: Moderate, foundational. An agent’s entire conversation history, retrieved documents, and tool results are all fed through positional encoding before the LLM processes them — this is precisely what lets the model distinguish “the user asked X, then the tool returned Y” from a scrambled, order-agnostic jumble of the same content.

Long agent conversations specifically stress-test a model’s positional encoding scheme, which is part of why RoPE’s better long-context generalization matters practically for agentic use cases.

When this knowledge is useful

Use Positional Information when you need to explain, implement, debug, evaluate, or optimize the corresponding part of a Transformer pipeline. It is also useful when a model API behaves unexpectedly and you need to trace the behavior back to tokens, tensor shapes, attention visibility, training, or inference mechanics.

When it is not enough

Understanding this mechanism does not by itself prove that a complete model or application is accurate, safe, fast, or cost-effective. Production decisions still require representative evaluation data, latency and memory measurements, model-specific documentation, and tests of the surrounding retrieval or agent code.

14. Common Beginner Mistakes

⚠️ Mistake

Incorrect idea: assuming attention “naturally” understands sequence order somehow.

Why it is incorrect: Section 9 proves the opposite directly — without positional information, attention output for the same word is identical regardless of sentence order.

⚠️ Mistake

Incorrect idea: treating positional encoding as a minor implementation detail.

Why it is incorrect: As demonstrated, its absence causes a complete failure to distinguish opposite-meaning sentences — this is a structurally essential component, not an optional refinement.

⚠️ Mistake

Incorrect idea: assuming RoPE and sinusoidal encoding work the same way, just with different formulas.

Why it is incorrect: They’re conceptually different: sinusoidal/learned encoding adds a positional vector to the embedding; RoPE rotates Q and K vectors, producing a fundamentally different (relative-position-aware) mathematical property.

15. Important Distinctions

Sinusoidal Positional Encoding	Learned Positional Embeddings
Fixed, computed via a formula, not trained	A trainable embedding table, learned via gradient descent
Same for every model using this scheme	Learned specifically for each model’s training data

Absolute Positional Encoding (sinusoidal/learned)	RoPE
Encodes each position independently, then adds it	Rotates Q/K based on position — dot product naturally reflects relative distance
No inherent “relative distance” property	Relative distance is a built-in mathematical consequence

16. Production / Engineering Considerations

Extending context length beyond a model’s original training range is a genuine, practical challenge — schemes like RoPE make certain extension techniques (interpolation/extrapolation of positional information) more tractable than fixed absolute encodings, a real factor in how some models support extended context windows after initial training.
Positional encoding choice is fixed at training time — like tokenization (Module 2), you cannot swap a trained model’s positional scheme without retraining.

17. Interview Questions

Beginner

Q: Why does a Transformer need positional information at all?

Ans: Self-attention computes relevance between tokens based purely on their content — it has no inherent sense of order. Two sentences using the same words in different orders would produce structurally identical attention computations without positional information, making the model unable to distinguish meaningfully different sentences like “dog bites man” from “man bites dog.”

Intermediate

Q: What’s the difference between sinusoidal positional encoding and learned positional embeddings?

Ans: Sinusoidal encoding is a fixed mathematical function (sine/cosine at varying frequencies) that generates a unique vector per position, computed the same way regardless of training data. Learned positional embeddings are a trainable table, structurally similar to token embeddings, with one row per position, whose values are learned via gradient descent specifically for that model’s training data.

Advanced

Q: What makes RoPE’s approach to positional information fundamentally different from adding a positional vector?

Ans: RoPE rotates the Query and Key vectors by an angle proportional to their position, rather than adding a separate positional vector to the token embedding. This means the dot product between a rotated Query and Key — the core of the attention score computation (Module 5) — mathematically reflects the relative distance between the two positions, not just their absolute positions independently.

Absolute encoding schemes (sinusoidal or learned) don’t have this built-in relative-distance property; the model has to learn to infer relative position indirectly from the added absolute positional information.

Scenario

Q: You remove positional encoding entirely from a working Transformer model and retrain it. What would you expect to happen to its ability to distinguish sentences that use the same words in different orders?

Ans: It would lose that ability almost entirely — as demonstrated directly in this module, self-attention without positional information produces identical output for the same words regardless of their order. The model would effectively treat any sentence as an unordered “bag of tokens,” unable to distinguish critically different meanings that depend on word order, such as subject-object relationships.

Architecture

Q: Why might a model trained with RoPE handle longer sequences at inference time better than one trained with fixed sinusoidal or learned absolute positional encoding?

Ans: RoPE encodes relative position directly in the attention score computation itself, rather than relying on the model to infer relative relationships from independently-added absolute positional vectors.

This relative-position property tends to generalize more naturally to sequence lengths beyond what was seen during training, compared to absolute schemes, which have no inherent mechanism for handling positions the model never encountered during training.

Engineering

Q: Why is positional encoding choice not something you can change after a model has been trained?

Ans: Like tokenization, positional encoding is baked into how the model learned to process its input during training — the model’s attention patterns and learned weights were optimized specifically around whatever positional information scheme was used.

Swapping schemes after training would present the model with positional signals it never learned to interpret correctly, requiring retraining (or at minimum significant fine-tuning) rather than a simple configuration change.

AI Engineering

Q: How does this module’s proof connect practically to something you’d debug in a real RAG or agent application?

Ans: If a system were somehow feeding unordered or incorrectly-ordered context into an LLM (e.g., a bug in how retrieved documents or conversation history get assembled into a prompt), the model would lose exactly the order-sensitivity this module demonstrates — potentially misinterpreting which statement came from the user versus a tool result, or losing track of causal/temporal relationships in a conversation.

This underscores why correctly ordering and assembling context before it reaches the LLM is a real, practical engineering concern, not just a theoretical one.

18. What You Should Remember

Self-attention is order-agnostic by itself — proven directly: identical token content in different orders produces byte-for-byte identical attention output without positional information.
Sinusoidal and learned positional embeddings add a positional vector to each token’s embedding; RoPE rotates Q/K vectors instead, giving the attention score a built-in relative-position property.
RoPE has become the common default for modern LLMs, largely due to its better generalization to longer sequences.

19. How This Helps Me Build AI Systems

You’ve now proven, not just been told, why positional information is structurally essential to every Transformer-based model — including every LLM you’ll build with.

This directly explains real, practical phenomena you’ll encounter: why context ordering matters when assembling prompts for RAG or agent systems, and why context-length limits and long-context performance are genuine architectural considerations, not arbitrary API restrictions.

Next: Module 9 — The Transformer Block — assembling attention, residual connections, LayerNorm, and the feed-forward network into the complete repeating unit every LLM is built from.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026