Attention – Deep Learning

Begin with the central question

What if a model could look directly at the most relevant earlier information instead of squeezing everything into one memory?

That question is the reason this topic exists. Keep it in mind as each new term appears: every equation, diagram, and code example below is one part of the answer.

query compares with keys → attention weights → weighted values

Before you continue: three tools for this module

Query: what the current position is looking for.
Key: what another position offers for comparison.
Value: the information contributed after relevance is calculated.

You do not need to memorize these definitions yet. Use them as a small map whenever the terms appear below.

What You Will Understand

This is one of the most important modules in the entire course. You’ll build attention from first principles — query, key, value, similarity scores, scaling, softmax, and the weighted sum — computing every step by hand and in code, until “what happens when a token attends to a sequence” is completely demystified.

Attention performs content-dependent information mixing:

query compares with keys → scores → scale and softmax → weights
values × weights → weighted sum for the current position

Query, key, and value are learned projections, not literal questions, labels, or database values. Standard self-attention connects positions directly and is parallelizable during training, but its compute and memory commonly grow quadratically with sequence length.

Why Each Position Needs Relevant Information from Other Positions

Module 14 showed RNNs’ two core limitations: gradients struggling over long sequences, and an inherently sequential computation pattern that resists parallelization. Attention exists to solve both at once: it lets every position in a sequence directly relate to every other position — regardless of distance — computed in parallel, with no recurrent hidden state passed step by step at all.

Relevance Scores and a Weighted Mixture

“When processing one piece of information, which other pieces should I pay attention to?” Attention answers this directly: for every token, it computes a relevance score against every other token in the sequence, then combines information from all of them, weighted by relevance — the more relevant a token is to the one currently being processed, the more it contributes.

⚠️ Do not imply attention means the model “consciously focuses” on something. It’s a precise mathematical weighting mechanism — a weighted average, computed via the steps below — not an act of conscious deliberation. The word “attention” is an evocative name, not a claim about awareness.

Analogy: The Job Search Matchmaker Agency Imagine you are a job recruiter matching candidates to open positions:

Query ( $Q$ - The Candidate’s Profile): A candidate comes in looking for a job. They say “I am a Python developer with 3 years of experience in ML” (their query). This represents what the current token is looking for.

Key ( $K$ - The Job Title Listing): You have a stack of open listings. One title says “React frontend engineer”, another says “PyTorch deep learning specialist”. These titles are the keys. They represent what each target token in the sequence has to offer.

Value ( $V$ - The actual Job Details): Behind each title listing is a detailed description containing the salary, location, and team structure (the values).

Similarity Score: You compare the candidate’s query ( $Q$ ) against all job listing keys ( $K$ ) using a similarity score (dot product $Q \cdot K^T$ ). The PyTorch listing gets a high score ( $0.95$ ), while React gets a low score ( $0.05$ ).

Softmax (Weights): You normalize these scores into probability weights that sum to 1.

Weighted Sum: You combine the details (Values $V$ ) of all jobs, scaled by their match weights. Since the PyTorch job has a $0.95$ weight, the candidate’s final matched offer contains mostly the PyTorch details, with a tiny contribution from the React details.

📊 Visual Flowchart: The Self-Attention Matrix Pipeline

Here is how token sequence vectors are projected, matched, scaled, and combined:

graph TD
    InputX["Input Sequence Embeddings (X)"] -->|Proj Wq| Q["Query Matrix (Q)"]
    InputX -->|Proj Wk| K["Key Matrix (K)"]
    InputX -->|Proj Wv| V["Value Matrix (V)"]

Q --> Dot["1. Similarity Check: Q @ K^T"]
    K --> Dot

Dot --> Scale["2. Scale: Divide by sqrt(d_k)"]
    Scale --> Softmax["3. Normalization: Softmax (row-wise)<br>(Yields Attention Weights Matrix)"]

Softmax --> WeightedSum["4. Weighted Context Sum:<br>Attention Weights @ V"]
    V --> WeightedSum

WeightedSum --> Output["5. Attention Output Matrix (Y)"]

4. Core Concept

Query, Key, Value

For every token, three separate vectors are computed via three learned projection matrices:

Query (Q):  "what am I looking for?" -- represents THIS token's
             current information need

Key (K):    "what do I contain?" -- represents what THIS token
             (as a potential source) has to offer

Value (V):  "what information do I actually provide?" -- the
             content that gets combined once relevance is decided

Q = X @ W_q
K = X @ W_k
V = X @ W_v

X is the matrix of input embeddings (Module 12) for every token in the sequence; W_q, W_k, W_v are learned weight matrices (Module 2’s familiar weighted-sum projections, applied here).

The complete flow

Query
  +
Keys
  ↓
Similarity scores    (Q @ K^T)
  ↓
Scaling               (divide by sqrt(d_k))
  ↓
Softmax                (Module 4 — converts scores into weights
                        that sum to 1)
  ↓
Attention weights
  ↓
Values
  ↓
Weighted sum           (attention_weights @ V)
  ↓
Attention output

5. How It Works — Step by Step

1. Compute Q, K, V for every token in the sequence (three
   separate learned linear projections of the input embeddings)
2. For each token's QUERY, compute its similarity to EVERY
   token's KEY: this is Q @ K^T -- a matrix where entry (i,j)
   is "how relevant is token j's key to token i's query?"
3. SCALE these raw scores by dividing by sqrt(d_k), where d_k
   is the key/query dimension -- this keeps the scores in a
   numerically well-behaved range before softmax (Section 6
   explains precisely why)
4. Apply SOFTMAX to each row -- converting each token's raw
   scores against every other token into a probability
   distribution that sums to 1 (Module 4)
5. Compute the WEIGHTED SUM of every token's VALUE vector,
   weighted by these attention weights -- this is the final
   attention output for that token

6. Mathematical Intuition

First, use only small numbers

If three attention scores become weights [0.1, 0.7, 0.2], the second item contributes 70% of the weighted mixture. Attention does not erase the others; it blends their value vectors in those proportions.

Read the mathematics as a story

Attention computes relevance scores, converts them into weights, and blends information accordingly. It creates a short direct path between related positions, even when they are far apart.

query compares with keys → attention weights → weighted values

Do not begin by memorizing the symbols. First identify what enters, what operation changes it, and what comes out. The symbols are a compact description of that journey.

Why scale by sqrt(d_k)? As the dimension d_k grows, the dot products Q @ K^T tend to grow larger in magnitude purely as a statistical consequence of summing more terms — and very large values fed into softmax push it toward an extremely “peaked” distribution (nearly all weight on one token, everything else near zero), which produces very small gradients during training (exactly Module 10’s saturation problem, applied here).

Dividing by sqrt(d_k) counteracts this growth, keeping softmax’s input in a numerically healthier range.

A complete, small worked example — 3 tokens (“cat”, “sat”, “mat”), each with a 4-dimensional embedding — computed entirely below in code, with every intermediate matrix shown.

7. Simple Example

Walk through the example

Read the example in three passes:

Identify the input numbers and what each number represents.
Follow one operation at a time instead of jumping directly to the answer.
Interpret the final number in ordinary language and connect it back to the problem.

The purpose is not merely to calculate the result. It is to make the internal mechanism visible. For the sentence “the cat sat,” when processing the word “sat,” its query vector gets compared against the keys of “the,” “cat,” and “sat” itself.

If “sat“‘s query is most similar to “cat“‘s key (a plausible outcome, since a verb’s grammatical subject is often highly relevant), “cat“‘s value vector will receive a much higher attention weight than “the“‘s — meaning “cat“‘s information contributes more heavily to “sat“‘s final attention output. This is what “attending to” a token concretely means: contributing more to the weighted sum, nothing more mystical.

8. Python Example

Three Python symbols used below

NumPy (np) is a Python library for working efficiently with lists and grids of numbers.
np.array(...) creates a numeric vector or matrix.
@ performs matrix multiplication: many connected weighted sums calculated together.

You can understand the concept without memorizing the syntax. First follow what the numbers represent, and then connect each code operation to the worked example.

What the code will demonstrate

Before running the code, predict the flow: create a small input, apply the topic’s calculation, and inspect the intermediate or final values. The example uses small numbers so you can connect each printed result to the explanation above; a real model performs the same kind of operation with much larger tensors and learned parameters.

# Build a tiny, inspectable example of Attention.
# Follow the intermediate values; they reveal what the model is doing.
import numpy as np

tokens = ["cat", "sat", "mat"]
X = np.array([
    [1.0, 0.0, 1.0, 0.0],   # "cat"
    [0.0, 1.0, 0.0, 1.0],   # "sat"
    [1.0, 1.0, 0.0, 0.0],   # "mat"
])

d_k = 4

np.random.seed(0)
W_q = np.round(np.random.randn(4, 4) * 0.5, 2)
W_k = np.round(np.random.randn(4, 4) * 0.5, 2)
W_v = np.round(np.random.randn(4, 4) * 0.5, 2)

Q = X @ W_q
K = X @ W_k
V = X @ W_v

print("Query matrix Q:\n", np.round(Q, 3))
print("\nKey matrix K:\n", np.round(K, 3))
print("\nValue matrix V:\n", np.round(V, 3))

# Step 1: similarity scores
scores = Q @ K.T
print("\nRaw similarity scores (Q @ K^T):\n", np.round(scores, 3))

# Step 2: scale
scaled_scores = scores / np.sqrt(d_k)
print("\nScaled scores:\n", np.round(scaled_scores, 3))

# Step 3: softmax (row-wise)
def softmax(x, axis=-1):
    exp_x = np.exp(x - np.max(x, axis=axis, keepdims=True))
    return exp_x / np.sum(exp_x, axis=axis, keepdims=True)

attention_weights = softmax(scaled_scores, axis=-1)
print("\nAttention weights (each row sums to 1):\n", np.round(attention_weights, 4))
print("Row sums:", attention_weights.sum(axis=1))

# Step 4: weighted sum of V
attention_output = attention_weights @ V
print("\nAttention output:\n", np.round(attention_output, 4))

print("\n--- What 'cat' (row 0) attends to ---")
for i, tok in enumerate(tokens):
    print(f"  attends to '{tok}': {attention_weights[0, i]:.4f}")

Expected Output:

Query matrix Q:
 [[ 0.83  0.41  0.56  1.85]
 [ 1.31 -0.43  0.7   0.09]
 [ 1.81 -0.29  0.97  1.04]]

Key matrix K:
 [[ 1.88 -0.83  0.18 -0.52]
 [-0.51  1.06  0.51 -0.18]
 [-0.53  0.23  0.59 -0.8 ]]

Value matrix V:
 [[-0.96 -1.7  -1.02  1.06]
 [ 0.37  0.38 -0.82  0.24]
 [ 0.18 -0.39 -0.36 -0.07]]

Raw similarity scores (Q @ K^T):
 [[ 0.359 -0.036 -1.495]
 [ 2.899 -0.783 -0.452]
 [ 3.277 -0.923 -1.286]]

Scaled scores:
 [[ 0.179 -0.018 -0.748]
 [ 1.449 -0.392 -0.226]
 [ 1.639 -0.461 -0.643]]

Attention weights (each row sums to 1):
 [[0.4512 0.3703 0.1785]
 [0.743  0.1179 0.1391]
 [0.8166 0.1    0.0834]]
Row sums: [1. 1. 1.]

Attention output:
 [[-0.264  -0.6959 -0.8281  0.5546]
 [-0.6446 -1.2726 -0.9046  0.8062]
 [-0.7319 -1.3828 -0.945   0.8838]]

--- What 'cat' (row 0) attends to ---
  attends to 'cat': 0.4512
  attends to 'sat': 0.3703
  attends to 'mat': 0.1785

9. How It Works

Every attention weights row sums to exactly 1 (confirmed by Row sums: [1. 1. 1.]) — softmax’s defining property, applied here per token.
“cat” attends to itself most strongly (0.4512), then “sat” (0.3703), then “mat” (0.1785) — a genuinely differentiated distribution, not a uniform average, computed purely from these (here, randomly initialized) W_q/W_k projections. In a trained model, these weight matrices would be learned so that grammatically/semantically relevant tokens receive systematically higher attention.
The final attention output for “cat” (row 0 of the output) is a weighted blend of all three tokens’ Value vectors, using exactly these weights — 0.4512 × V[cat] + 0.3703 × V[sat] + 0.1785 × V[mat] — this is the literal computation producing each token’s new, context-aware representation.

10. Real-World Example

When an LLM processes the sentence “The trophy didn’t fit in the suitcase because it was too big,” resolving what “it” refers to requires relating “it” back to “trophy” (not “suitcase”) — a long-range, context-dependent relationship.

Self-attention computes exactly this kind of relevance directly, in one step, regardless of how many words separate “it” from “trophy” — precisely the long-range dependency problem Module 14 showed RNNs struggling with.

11. Self-Attention vs. Cross-Attention

Self-attention:    Q, K, and V all come from the SAME sequence
                    -- a sequence relating to itself (e.g., every
                    word in a sentence relating to every other
                    word in that same sentence)

Cross-attention:    Q comes from ONE sequence, K and V come from
                    a DIFFERENT sequence -- e.g., a decoder's
                    query attending to an encoder's output
                    (Module 16 covers exactly where this appears)

Section 8’s example is self-attention — Q, K, and V were all derived from the same input X.

12. How Is This Used in Modern AI?

Follow it from mechanism to product

Transformer LLMs apply attention across token representations in every block. Attention weights show one information-routing mechanism, but they are not a complete or guaranteed explanation of a model’s reasoning.

How this connects to LLMs

prompt → tokens → deep-learning computations → next-token probabilities → generated response

The model computation is only the middle of the journey. Tokenization happens before it, while decoding and application controls happen afterward; the following example identifies this topic’s exact role.

🤖 Real-world connection

Self-attention is a core mechanism in Transformer language models (Module 16), which underpin the dominant LLM architectures today. When these models process a prompt, variants of the Q/K/V, score, normalization, and weighted- value computation run across their attention layers. Implementations can use multi-query, grouped-query, sparse, linearized, or other attention variants, so not every model performs the identical textbook calculation everywhere.

Concept	AI application
Self-attention	A core mechanism in Transformer-based LLM blocks
Softmax over attention scores	Directly reuses Module 4’s softmax, applied here to relevance scores instead of classification logits
Cross-attention	Used in encoder-decoder architectures (e.g., translation models — Module 16)
Attention weights	Sometimes visualized directly to interpret what a model “focused on” for a given output — a genuinely useful, if imperfect, interpretability tool

13. How Is This Used in Agentic AI?

Trace one agent step

goal + history + tool results → LLM proposal → runtime validation → tool or response

The deep-learning model produces a prediction or structured proposal. The agent runtime—ordinary software around the model—controls permissions, executes tools, stores state, handles retries, and decides whether another model call is needed.

Direct relevance to Agentic AI: Very High, entirely through the LLM that powers the agent’s reasoning.

Every contextual understanding an agent’s LLM demonstrates — connecting a user’s follow-up question back to something mentioned several turns earlier, correctly resolving which prior tool result a new instruction refers to — is, mechanically, self-attention computing exactly the relevance-weighted combination demonstrated in Section 8, just at the scale of a full conversation history instead of a 3-token toy sentence.

14. Common Beginner Mistakes / Misconceptions Corrected

⚠️ Mistake

Incorrect idea: “attention” implies conscious focus or awareness.

Why it is incorrect: As stated in Section 3 — it’s a precise, mechanical weighting computation. The name is evocative, not literal.

⚠️ Mistake

Incorrect idea: Q, K, and V are three unrelated, independently meaningful vectors.

Why it is incorrect: They’re all derived from the same input embeddings via three different learned linear projections — their distinct “roles” (query/key/value) emerge from how they’re used in the attention computation, not from any inherent property.

⚠️ Mistake

Incorrect idea: “Transformers are just attention.”

Why it is incorrect: Attention is the defining, novel mechanism, but a full Transformer block also includes feed-forward layers, residual connections, and normalization (Module 16) — attention alone is one critical component, not the whole architecture.

15. Important Distinctions

Self-Attention	Cross-Attention
Q, K, V all from the same sequence	Q from one sequence, K/V from another
A sequence relating to itself	One sequence relating to a different sequence

Attention (general mechanism)	Self-Attention (specific case)
The broader query/key/value weighting mechanism	Attention specifically where Q, K, V share a source

16. When to Use

Self-attention is the standard mechanism for any task requiring context-aware relationships across a sequence — essentially all modern NLP and increasingly vision/audio tasks. Cross-attention specifically applies when relating two distinct sequences (e.g., a question and a document, in some architectures).

17. When Not to Use

Not applicable in the traditional “when not to use” sense — attention (in some form) is a near-universal component of modern sequence models. The more relevant question is architectural: whether a full Transformer (Module 16) is warranted versus a simpler model, which depends on task complexity and available data/compute, not on attention itself being inappropriate.

18. Interview Questions

Beginner

Q: What problem does attention solve?

Ans: It lets every position in a sequence directly relate to every other position, regardless of distance, computed in parallel — solving both of RNNs’ core limitations (Module 14) at once: vulnerability to vanishing gradients over long sequences, and an inherently sequential computation that resists parallelization.

Intermediate

Q: What are Query, Key, and Value, and how do they relate to each other?

Ans: They’re three separate vectors computed for each token via three different learned linear projections of the same input embedding. The Query represents what a token is “looking for”; the Key represents what a token “offers” as a potential match; the Value is the actual information contributed once relevance (via Query-Key similarity) is determined.

All three come from the same underlying embedding — their distinct roles come from how they’re used in the attention computation.

Advanced

Q: Why is the scaling step (dividing by sqrt(d_k)) necessary in scaled dot-product attention?

Ans: As the key/query dimension d_k grows, the raw dot products Q @ K^T tend to grow larger in magnitude, purely because more terms are being summed.

Very large values fed into softmax push it toward an extremely peaked distribution — nearly all weight on one position — which produces very small gradients during backpropagation (analogous to the activation saturation problem in Module 4/10), slowing or destabilizing training.

Dividing by sqrt(d_k) counteracts this dimension-dependent growth, keeping the softmax input in a numerically well-behaved range regardless of d_k.

Scenario

Q: You’re debugging a Transformer-based model and want to understand why it produced a particular output for a specific token. How could attention weights help, and what’s a real limitation of this approach?

Ans: Visualizing the attention weights for that token — exactly the kind of matrix computed in Section 8 — shows which other tokens it weighted most heavily, offering a genuine, if partial, interpretability signal.

The real limitation: a modern Transformer has many attention heads across many layers (Module 16), and a single attention weight matrix from one head/layer doesn’t capture the model’s full, distributed reasoning process — attention weights are a useful diagnostic clue, not a complete explanation of model behavior.

AI Engineering

Q: How does self-attention directly enable an LLM to handle long-range context, like resolving a pronoun to a noun mentioned many sentences earlier?

Ans: Self-attention computes a direct relevance score between every pair of positions in a sequence — including a pronoun near the end and a noun near the beginning — in a single computation, with no need to pass information sequentially through every intermediate position (unlike an RNN, Module 14).

If the model has learned that this pronoun’s query is meaningfully similar to that noun’s key, the noun’s value contributes significantly to the pronoun’s attention output, regardless of how many tokens separate them in the sequence.

19. What You Should Remember

Attention computes, for every token: a Query (“what am I looking for”), compared against every token’s Key (“what do I offer”), to produce relevance scores — scaled, passed through softmax, and used to compute a weighted sum of every token’s Value.
Scaling by sqrt(d_k) keeps softmax numerically well-behaved regardless of dimension size.
Self-attention (Q/K/V from the same sequence) is the core mechanism inside every Transformer block — verified here with a complete, real computation, not just a description.

20. How This Helps Me Build AI Systems

You’ve now computed a complete attention operation by hand, with real matrices — Query, Key, Value, scores, scaling, softmax, weighted sum — exactly the computation that runs, many times over, every time an LLM processes anything. Module 16 assembles this mechanism into the complete Transformer block; nothing new needs to be introduced there beyond what you’ve already computed here.

Next: Module 16 — Transformers — multi-head attention, residual connections, layer normalization, and the complete Transformer block, including why GPT-style LLMs are decoder-only.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026