Self-Attention From First Principles

Begin with the central question

How can the word being processed directly collect useful information from every other word?

Essential words

Self-attention relates positions within the same sequence. A score estimates relevance. A weighted sum mixes information according to normalized scores.

What You Will Understand

This is one of the most important modules in the entire course. You’ll build self-attention from raw intuition — starting with “a token needs to understand which other tokens are relevant to it” — using real numbers on a tiny 4-token sentence, computed step by step, before the formal Query/Key/Value formulation (Module 4) is introduced at all.

one token -> compare with all tokens -> weights -> weighted context

The problem this module solves

Module 2 showed you what enters a Transformer: a sequence of vectors, one per token, each just sitting there independently.

But a word’s meaning in context often depends heavily on the other words around it — “bank” means something different next to “river” than next to “loan.” Self-attention exists to let each token’s representation be updated based on the other tokens it’s actually relevant to, before any further processing happens.

Build the intuition

“A token needs to understand which other tokens are relevant to it.” Self-attention answers this directly: for a given token, measure how related it is to every other token in the sequence (including itself), turn those relatedness scores into weights, and blend all the tokens together using those weights — tokens that are more relevant contribute more to the new, “contextualized” version of that token.

4. Real-World Analogy

Imagine you’re trying to understand the word “tired” in the sentence “the animal was tired.” To really grasp what’s tired, you don’t just look at the word “tired” in isolation — you glance back at “animal” (the thing that’s tired) far more than at “the” or “was” (which don’t add much meaning here).

You’re not ignoring the other words entirely — you’re weighting your attention toward the ones that matter most for understanding this particular word.

Analogy: The Contextual Word Highlighter (Relevance Arrows) Think of self-attention as drawing highlighting arrows across a printed sentence:

The Problem: A dictionary entry has static definitions, but words are context-dependent chameleon shapes: “The bank was muddy near the river” versus “The bank approved my mortgage loan.”

The Highlighter (Self-Attention):

For target word “bank”, you scan the page and measure compatibility against every other word.

In the first sentence, your brain highlights “river” and “muddy” with bright markers (high dot-product similarity) and leaves “the” and “was” unhighlighted.

In the second sentence, your highlighter marks “loan” and “mortgage” instead.

The new “context-aware definition” of “bank” is a blended memory: 70% “bank” itself + 25% “river” + 5% “muddy”. The noun adapts directly to its physical surroundings.

📊 Visual Flowchart: Self-Attention Matrix Dot-Product Blend

Here is how similarity scores compile into row-wise softmax weights to output blended contextual vectors:

graph TD
    EmbedMatrix["Input Embedding Matrix (Sequence Length x d_model)<br>['the', 'animal', 'was', 'tired']"] --> SimScores["1. Similarity Score Matrix (Seq Length x Seq Length)<br>(Compute raw dot products: X @ X.T)"]




    SimScores --> Softmax["2. Row-wise Softmax Normalization<br>(Each row becomes positive weights summing to 1.0)"]




    subgraph MatrixMultiply ["Weighted Sum Synthesis"]
        Softmax --> WeightedMultiply["3. Dot Product Weighting Matrix<br>(weights @ X)"]
        EmbedMatrix --> WeightedMultiply
    end




    WeightedMultiply --> ContextVectors["4. Output Contextual Embedding Matrix<br>(Position-aware contextual representations)"]

5. Core Concept

For a sequence of token embeddings (Module 2), self-attention computes, for every token, a new representation that’s a weighted blend of every token in the sequence — where the weights reflect how relevant each token is to the one being updated.

For token X:
  1. Measure how RELEVANT every token (including X itself) is to X
  2. Turn those relevance measurements into WEIGHTS that sum to 1
  3. Blend every token's embedding together, using those weights
  4. The result is X's NEW, context-aware representation

🧠 You already saw attention conceptually in Deep Learning Module 15. Here, the goal is to build the intuition fully from scratch with a concrete example, before Module 4 formalizes it with the standard Query/Key/Value machinery.

6. How It Works — Step by Step

1. Start with token embeddings (Module 2's output) -- one vector
   per token
2. For a given token, compute a RAW SIMILARITY SCORE between it
   and EVERY token in the sequence (a simple dot product is
   enough to build the intuition -- higher dot product = more
   similar/aligned vectors)
3. Convert these raw scores into ATTENTION WEIGHTS using softmax
   -- this ensures the weights are all positive and sum to
   exactly 1, so they behave like a genuine "how much to blend
   in" percentage
4. Compute the WEIGHTED SUM of every token's embedding, using
   these attention weights -- this produces the token's NEW,
   contextual representation
5. Repeat this ENTIRE process independently for every token in
   the sequence

7. Mathematical Intuition

Read the mathematics as a story

Attention performs three understandable actions: compare, normalize, and mix. A score says how strongly two representations match; softmax turns scores into shares; the weighted sum collects information.

compare token pairs -> scores -> softmax weights (sum = 1) -> weighted mixture

A dot product between two vectors is large and positive when they point in similar directions (Module 12 of the Deep Learning course covered this for embeddings generally) — a simple, effective first measure of “how related are these two tokens’ representations.” Softmax (DL Module 4) then converts these raw, unbounded scores into a genuine probability- like distribution — weights that are all positive and sum to 1, suitable for a weighted blend.

8. Small Worked Example

Walk through the example

Focus on one token. 2. Give it a score against every token. 3. Normalize the scores. 4. Multiply each token vector by its weight. 5. Add the results.

Take the sentence “the animal was tired” — 4 tokens. Focus on the last token, “tired.” Intuitively, “tired” should be most related to itself and to “animal” (the thing that’s actually tired) — much more than to “the” or “was,” which carry little independent meaning here. This is exactly what gets computed and verified below.

9. Python / NumPy Example

What the code will demonstrate

This small NumPy example makes Self-Attention From First Principles visible with inspectable numbers and shapes. Read it in three passes: identify each input, follow the transformation line by line, and connect the printed output to the diagram above. The arrays are intentionally tiny teaching values; unless the text explicitly says otherwise, they are not weights or measurements from GPT, Gemini, or another trained model.

# The arrays are intentionally small so each transformation can be inspected.
# Printed values illustrate the mechanism; they are not trained-model measurements.
import numpy as np




def softmax(x):
    exp_x = np.exp(x - np.max(x))
    return exp_x / np.sum(exp_x)




# 4 tokens: "the animal was tired"
# Hand-crafted embeddings (illustrative, not learned) where semantically
# related words are deliberately closer together in vector space --
# exactly like real TRAINED embeddings end up (DL Module 12)
tokens = ["the", "animal", "was", "tired"]
X = np.array([
    [0.1, 0.1, 0.0, 0.0],   # "the" -- a function word, fairly neutral
    [0.9, 0.8, 0.1, 0.2],   # "animal" -- a concrete noun
    [0.1, 0.0, 0.9, 0.1],   # "was" -- a function/linking word
    [0.8, 0.7, 0.2, 0.9],   # "tired" -- an adjective, related to "animal"
])
print("Token embeddings:\n", X)




# --- Step 1: relevance of every token to "tired" (raw dot product) ---
query_word = "tired"
query_idx = tokens.index(query_word)
query_vector = X[query_idx]




raw_scores = X @ query_vector
print(f"\nRelevance of every token to '{query_word}':")
for tok, score in zip(tokens, raw_scores):
    print(f"  '{tok}' . '{query_word}' = {score:.4f}")




# --- Step 2: attention weights via softmax ---
attention_weights = softmax(raw_scores)
print("\nAttention weights (sum to 1):")
for tok, w in zip(tokens, attention_weights):
    print(f"  '{tok}': {w:.4f}")
print("Sum:", attention_weights.sum())




# --- Step 3: weighted sum -> new contextual representation ---
contextual_representation = attention_weights @ X
print(f"\nNew, CONTEXTUAL representation for '{query_word}':")
print(" ", np.round(contextual_representation, 4))
print(f"\nOriginal '{query_word}' embedding was:", X[query_idx])

Expected Output:

Token embeddings:
 [[0.1 0.1 0.  0. ]
 [0.9 0.8 0.1 0.2]
 [0.1 0.  0.9 0.1]
 [0.8 0.7 0.2 0.9]]




Relevance of every token to 'tired':
  'the' . 'tired' = 0.1500
  'animal' . 'tired' = 1.4800
  'was' . 'tired' = 0.3500
  'tired' . 'tired' = 1.9800




Attention weights (sum to 1):
  'the': 0.0817
  'animal': 0.3090
  'was': 0.0998
  'tired': 0.5095
Sum: 1.0




New, CONTEXTUAL representation for 'tired':
  [0.7038 0.612  0.2226 0.5303]




Original 'tired' embedding was: [0.8 0.7 0.2 0.9]

Exactly as intuition predicted: animal received the second-highest attention weight (0.3090), well above the (0.0817) and was (0.0998) — “tired” genuinely attends most to itself, then to the noun it’s actually describing. The new contextual representation is noticeably different from the original raw embedding — it now carries blended-in information from “animal,” not just “tired” in isolation.

Now the full attention matrix — every token attending to every other token, all at once:

# The arrays are intentionally small so each transformation can be inspected.
# Printed values illustrate the mechanism; they are not trained-model measurements.
def softmax_rows(x):
    exp_x = np.exp(x - np.max(x, axis=-1, keepdims=True))
    return exp_x / np.sum(exp_x, axis=-1, keepdims=True)




all_scores = X @ X.T
print("Raw score matrix (every token vs every token):\n", np.round(all_scores, 3))




all_weights = softmax_rows(all_scores)
print("\nAttention weight matrix (each ROW sums to 1):\n", np.round(all_weights, 4))
print("Row sums:", all_weights.sum(axis=1))




contextual_all = all_weights @ X
print("\nAll tokens' new contextual representations:\n", np.round(contextual_all, 4))




for i, tok in enumerate(tokens):
    top = np.argmax(all_weights[i])
    print(f"'{tok}' attends most strongly to '{tokens[top]}' ({all_weights[i][top]:.4f})")

Expected Output:

Raw score matrix (every token vs every token):
 [[0.02 0.17 0.01 0.15]
 [0.17 1.5  0.2  1.48]
 [0.01 0.2  0.83 0.35]
 [0.15 1.48 0.35 1.98]]




Attention weight matrix (each ROW sums to 1):
 [[0.2331 0.2708 0.2307 0.2654]
 [0.1051 0.3973 0.1083 0.3894]
 [0.1699 0.2055 0.3858 0.2387]
 [0.0817 0.309  0.0998 0.5095]]
Row sums: [1. 1. 1. 1.]




All tokens' new contextual representations:
 [[0.5024 0.4257 0.2878 0.3161]
 [0.6904 0.6009 0.215  0.4407]
 [0.4315 0.3485 0.4155 0.2946]
 [0.7038 0.612  0.2226 0.5303]]




'the' attends most strongly to 'animal' (0.2708)
'animal' attends most strongly to 'animal' (0.3973)
'was' attends most strongly to 'was' (0.3858)
'tired' attends most strongly to 'tired' (0.5095)

How It Works

Every row of the attention weight matrix sums to exactly 1.0 — softmax’s defining property, applied per token, confirmed directly.
Notice 'the' attends most strongly to 'animal', not to itself — makes intuitive sense, since “the” alone carries very little meaning, and borrowing context from the nearby noun genuinely enriches its representation.
'animal' and 'tired' both attend most strongly to themselves, but with their second-highest weight going to each other — exactly the semantic relationship this example was designed (via the hand-crafted embeddings) to surface, now computed mechanically rather than assumed.

10. How Is This Used in Modern AI?

Where this concept lives

Follow the concept at three levels: inside the model, where the computation happens; inside the AI product, where that computation supports a visible feature; and inside production, where engineers measure speed, memory, quality, and failure cases. The details below connect those levels.

🤖 How Is This Used in Modern AI?

This exact mechanism — relevance scores, softmax, weighted blend — is the computational core inside every attention layer of every modern Transformer, including every LLM. What you’ve just computed by hand is the simplified version (using raw embeddings directly); Module 4 introduces the refinement (separate learned Query, Key, and Value projections) that real Transformers actually use, but the underlying intuition — “compute relevance, weight, blend” — doesn’t change.

11. How Is This Used in LLMs?

Trace one model call

User text → tokens → Transformer computation → output-token probabilities
          this topic affects one part of that computation

An LLM does not apply this idea as a separate magic step. It uses it as part of the repeated numerical pipeline that transforms token vectors and produces the next-token probabilities.

Every LLM’s self-attention layers compute exactly this kind of relevance-weighted blending, at every layer, for every token, using learned (not hand-crafted) embeddings and learned projection weights (Module 4).

This is the literal mechanism by which an LLM “understands” that a pronoun refers to a noun mentioned earlier, or that an adjective describes a specific subject — contextual understanding is not a separate module bolted onto the model; it’s this weighted-blending computation, repeated across many layers.

Real systems you can recognize

Transformer-based GPT and Gemini systems use attention-family mechanisms to build contextual states. Hugging Face can expose per-layer attention tensors when a compatible model is called with output_attentions=True; see model outputs.

12. How Is This Used in Agentic AI?

Trace one agent step

Goal + history + tool results
              ↓
     LLM processes the context
              ↓
Suggested answer or tool call
              ↓
Agent runtime validates and executes it

This distinction matters: the Transformer helps produce the proposal, while the surrounding agent software controls tools, permissions, retries, memory, and execution.

Direct relevance to Agentic AI: High, indirectly through the LLM. Every time an agent’s LLM correctly connects a user’s follow-up question to something mentioned several turns earlier, or resolves which prior tool result a new instruction is referring to, this exact relevance-weighting mechanism is what’s happening underneath — computed across the agent’s full assembled context, not just a 4-token toy sentence.

When this knowledge is useful

Use Self-Attention From First Principles when you need to explain, implement, debug, evaluate, or optimize the corresponding part of a Transformer pipeline. It is also useful when a model API behaves unexpectedly and you need to trace the behavior back to tokens, tensor shapes, attention visibility, training, or inference mechanics.

When it is not enough

Understanding this mechanism does not by itself prove that a complete model or application is accurate, safe, fast, or cost-effective. Production decisions still require representative evaluation data, latency and memory measurements, model-specific documentation, and tests of the surrounding retrieval or agent code.

13. Common Beginner Mistakes

⚠️ Mistake

Incorrect idea: assuming “attention” means conscious focus.

Why it is incorrect: As covered in DL Module 15 — it’s a precise mathematical weighting computation, not an act of deliberation. The name is evocative, not literal.

⚠️ Mistake

Incorrect idea: thinking each token only attends to nearby tokens.

Why it is incorrect: Self-attention computes relevance between every pair of tokens in the sequence, regardless of distance — “the” in this example attended most strongly to “animal,” not to its immediate neighbor.

⚠️ Mistake

Incorrect idea: believing this simplified raw-embedding version is exactly what real Transformers use.

Why it is incorrect: It captures the core intuition correctly, but real self-attention uses three separate learned projections (Query, Key, Value) instead of using the raw embeddings directly for everything — Module 4 explains precisely why that separation matters.

14. Important Distinctions

Attention Score	Attention Weight
The RAW relevance measurement (e.g., a dot product)	The score AFTER softmax — normalized, positive, sums to 1
Can be any real number	Always between 0 and 1

Original Embedding	Contextual Representation
A token’s meaning in isolation (Module 2)	A token’s meaning AFTER blending in relevant context via attention

15. Production / Engineering Considerations

Computing the full attention matrix requires comparing every token against every other token — for a sequence of length n, this means n × n comparisons. This quadratic relationship is exactly why very long sequences become computationally expensive (Module 17 covers this precisely) — a genuine, practical engineering constraint that traces directly back to this module’s mechanism.

16. Interview Questions

Beginner

Q: What problem does self-attention solve?

Ans: It lets each token’s representation be updated based on which other tokens in the sequence are actually relevant to it, producing a context-aware representation instead of relying on the token’s in-isolation meaning alone.

Intermediate

Q: Walk through the three main steps of computing self-attention for one token.

Ans: First, compute a raw relevance score between that token and every token in the sequence (including itself) — a dot product is a simple, effective measure. Second, convert these raw scores into attention weights using softmax, ensuring they’re all positive and sum to 1. Third, compute the weighted sum of every token’s embedding using these weights, producing the token’s new, contextual representation.

Advanced

Q: Why does self-attention compute relevance between every pair of tokens, rather than only nearby ones?

Ans: Because relevant relationships in language aren’t limited by distance — a pronoun can refer to a noun many words earlier, or a verb’s meaning can depend on a subject mentioned much earlier in a long sentence.

Restricting attention to only nearby tokens would reintroduce exactly the long-range dependency problem that motivated moving away from recurrent architectures in the first place (Module 1) — computing full pairwise relevance, regardless of distance, is what gives attention its advantage here.

Scenario

Q: In the worked example, “the” ended up attending most strongly to “animal” rather than to itself. Is this a bug, or expected behavior? Explain.

Ans: This is expected, sensible behavior, not a bug. “The” alone carries very little independent meaning — a determiner whose interpretation depends entirely on what it’s referring to. Attending strongly to “animal” (the noun it precedes and refers to) lets “the“‘s new representation absorb genuinely useful contextual information, rather than remaining an uninformative, generic vector.

This kind of context-borrowing is precisely the value self-attention adds over using raw, context-free embeddings alone.

Architecture

Q: What ensures attention weights are valid for a weighted sum (i.e., that they behave like proper blending percentages)?

Ans: The softmax function — it guarantees every weight is positive and that all weights for a given token sum to exactly 1, verified directly in this module’s example (Row sums: [1. 1. 1. 1.]). Without this normalization step, raw similarity scores could be negative or of wildly different magnitudes, making a “weighted blend” nonsensical.

AI Engineering

Q: How does understanding this mechanism help you reason about why an LLM sometimes seems to “miss” a relevant piece of earlier context in a long conversation?

Ans: Since attention weights are a softmax-normalized distribution across the entire context, adding much more content to the context effectively means the same “attention budget” (weights summing to 1) gets spread across more competing tokens — highly relevant information can end up with a smaller share of attention simply because there’s more content competing for it, not because the mechanism is fundamentally broken.

This is a genuine, mechanism-level intuition for why very long, noisy contexts can sometimes dilute a model’s effective use of the most relevant pieces of information within them.

17. What You Should Remember

Self-attention: for every token, measure relevance to every other token (including itself), convert to weights via softmax, then compute a weighted blend — producing a new, contextual representation.
Verified directly: attention weights genuinely sum to 1, and the resulting contextual representations meaningfully differ from the original embeddings, blending in relevant information from related tokens.
This is the simplified, intuition-first version — Module 4 adds the Query/Key/Value refinement real Transformers actually use.

18. How This Helps Me Build AI Systems

You’ve now computed self-attention’s core intuition entirely by hand, with real numbers that behaved exactly as expected — “tired” genuinely attending to “animal.” Every subsequent module in this course refines this same core idea (adding Q/K/V, scaling, masking, multiple heads), never replacing it. If you understand this module deeply, everything that follows is elaboration, not a new concept.

Next: Module 4 — Query, Key and Value — why real Transformers use three separate learned projections instead of the raw embeddings directly, and exactly how each one is used.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026