Why Transformers? | TechByteByByte

Begin with the central question

Why did reading words one at a time become a bottleneck for large language models?

Essential words

A sequence is an ordered list of tokens. Recurrence processes positions one after another. A Transformer uses attention-based blocks so training can process many positions in parallel.

What You Will Understand

The specific, concrete technical problems that RNNs and LSTMs had — not just “they were worse” — and exactly how attention and the Transformer architecture solved each one. By the end, the historical progression from RNN to LLM will feel like a chain of well-motivated engineering decisions, not an arbitrary sequence of trends.

tokens -> parallel attention paths -> contextual token states

The problem this module solves

You already covered RNNs, LSTMs, and attention conceptually in the Deep Learning course (DL Module 14-15). This module isn’t a re-teach — it’s the specific “why” that connects that material to everything the rest of this course builds: exactly what problem the Transformer architecture was designed to solve, and why that design choice turned out to matter so much more than anyone expected in 2017.

Build the intuition

an RNN reads a sentence the way a person reads with their finger moving one word at a time, unable to skip ahead or look at two words simultaneously. A Transformer reads the way a person can glance at an entire page at once, instantly relating any word to any other word, regardless of distance or order of reading.

4. Real-World Analogy

Think of a group project with a strict rule: each team member can only start their part after the previous person finishes and hands off their work (an RNN’s step-by-step dependency). Now imagine instead every team member can read the entire shared document simultaneously and contribute based on the whole picture at once — nobody waits on anyone else (attention’s parallel, whole-sequence view).

The second team finishes dramatically faster, especially as the team (or sentence) grows.

Analogy: The Group Project Relay Race vs. The Collaborative Shared Doc Think of recurrent models versus attention architectures in terms of project workflow coordination:

The Relay Race (RNN/LSTM): Imagine writing a 100-page group report where teammate 4 cannot write a single word until teammate 3 hand-delivers their paper. Teammate 3 cannot start until teammate 2 finishes. If the team size grows to 100, teammate 100 sits idle for hours. The processing time grows linearly with the length of the document.

The Shared Doc (Transformer Attention): All 100 teammates open a single shared Google Doc simultaneously. Everyone looks at any paragraph instantly, reads the entire workspace, and writes their sections in parallel. Nobody sits waiting.

Because there are no sequential time steps, you can harness 1,000 GPU cores to compute all 100 page relationships in a fraction of a second.

📊 Visual Chart: Sequential Recurrence vs. Parallel Attention Grid

Here is why RNN loops stall computation while Transformers compute all positions simultaneously:

graph TD
    subgraph RNN ["Recurrent Network (Sequential Bottleneck)"]
        X0["Token 0"] --> H0["State 0"]
        X1["Token 1"] --> H1{"State 1<br>(Blocked until State 0 is ready)"}
        H0 --> H1
        X2["Token 2"] --> H2{"State 2<br>(Blocked until State 1 is ready)"}
        H1 --> H2
    end




    subgraph Attention ["Attention Grid (Fully Parallel)"]
        AX0["Token 0"] -.-> AScore["Compute all Query-Key dot products in parallel"]
        AX1["Token 1"] -.-> AScore
        AX2["Token 2"] -.-> AScore
        AScore --> Out["Blended Output Representation Matrix"]
    end

5. Core Concept — Two Distinct Problems, Not One

RNNs (DL Module 14) had two genuinely separate limitations. Conflating them is a common mistake — they need distinct fixes.

Problem	What it means	Fixed by
Vanishing gradients over long sequences	Gradients shrink exponentially across many time steps, so early tokens’ influence is lost	LSTM/GRU gating helped, but didn’t eliminate it (DL Module 14)
Sequential computation (no parallelization)	Step `t` cannot be computed until step `t-1` finishes — a hard, structural dependency	Solved by attention, which has no such dependency at all

🧠 LSTM/GRU meaningfully addressed the first problem. Neither addressed the second — and as training data and model sizes grew, the inability to parallelize became the more decisive, practical bottleneck.

6. How It Works — Step by Step: The Historical Progression

RNN
  ↓  (problem: no memory of earlier tokens beyond a few steps;
      severe vanishing gradients)
LSTM / GRU
  ↓  (problem: gating helps memory, but every step STILL must
      wait for the previous step to finish -- fundamentally
      sequential, resists parallelization)
Attention
  ↓  (breakthrough: relate any two tokens directly, with NO
      step-by-step dependency at all)
Transformer (2017)
  ↓  (wraps attention with the supporting architecture needed
      to train it well at depth -- Modules 9-11 of this course)
BERT / GPT
  ↓  (large Transformer models, pretrained at scale)
Large Language Models
  ↓  (further scaled, on vastly more data and parameters)
Generative AI
  ↓  (LLMs applied to open-ended content generation)
Agentic AI
  ↓  (LLMs wrapped with tools, memory, and orchestration —
      Module 18 of this course covers this transition precisely)

Each arrow represents a genuine engineering response to a specific, identifiable limitation — not a trend followed for its own sake.

7. Mathematical Intuition

Read the mathematics as a story

An RNN creates a dependency chain: h₃ waits for h₂, which waits for h₁. Attention compares positions from the available input states, so those pair comparisons do not wait for earlier pair results.

RNN: x₀ -> h₀ -> h₁ -> h₂ -> h₃
Attention: (x₀,x₁)  (x₀,x₂)  (x₀,x₃) ... can be computed together

The core structural fact, stated precisely: an RNN’s hidden state computation is h[t] = f(x[t], h[t-1]) — by definition, h[t] cannot be computed without already having h[t-1].

Attention’s raw relevance computation between any two positions i and j has no such dependency — computing the relationship between positions 3 and 4 needs neither position 1’s nor position 2’s result, only the raw input data itself, which is available immediately for the entire sequence.

8. Small Worked Example

Walk through the example

Use five token vectors. 2. Compute RNN states in forced order. 3. Compute attention-score rows in shuffled order. 4. Confirm the attention scores do not change.

Consider a 5-token sequence. An RNN’s hidden state at position 3 needs position 2’s hidden state, which needs position 1’s, which needs position 0’s — a strict, forced ordering of computation. Attention’s raw similarity score between position 3 and position 0 requires only their original input vectors — it could, in principle, be computed before, after, or simultaneously with any other pair’s score.

This isn’t a minor implementation detail — it’s demonstrated concretely below.

9. Python / NumPy Example

What the code will demonstrate

This small NumPy example makes Why Transformers? visible with inspectable numbers and shapes. Read it in three passes: identify each input, follow the transformation line by line, and connect the printed output to the diagram above. The arrays are intentionally tiny teaching values; unless the text explicitly says otherwise, they are not weights or measurements from GPT, Gemini, or another trained model.

# The arrays are intentionally small so each transformation can be inspected.
# Printed values illustrate the mechanism; they are not trained-model measurements.
import numpy as np




np.random.seed(0)
seq_len = 5
d = 4




X = np.random.randn(seq_len, d)
Wh = np.random.randn(d, d) * 0.3




# RNN-style: hidden state at step t REQUIRES step t-1's result first.
# This is a genuine, structural dependency -- step 3 cannot be computed
# until step 2 finishes, no matter how much parallel hardware is available.
def rnn_forward(X, Wh):
    h = np.zeros(len(Wh))
    states = []
    for t in range(len(X)):
        h = np.tanh(X[t] + h @ Wh)   # depends on h from the PREVIOUS iteration
        states.append(h.copy())
    return np.array(states)




rnn_states = rnn_forward(X, Wh)
print("RNN hidden states (each REQUIRES the previous one):\n", np.round(rnn_states, 3))




# Attention-style: every position's raw similarity score can be computed
# independently -- no position needs another position's RESULT first.
# Proof: compute scores in normal order, then in a SHUFFLED order --
# identical result either way.
def attention_scores_normal_order(X):
    return X @ X.T




def attention_scores_out_of_order(X):
    n = len(X)
    scores = np.zeros((n, n))
    row_order = [3, 0, 4, 1, 2]   # deliberately non-sequential
    for i in row_order:
        scores[i] = X[i] @ X.T   # row i needs ONLY X[i] and all of X
    return scores




scores_normal = attention_scores_normal_order(X)
scores_shuffled = attention_scores_out_of_order(X)




print("\nAttention scores (normal order):\n", np.round(scores_normal, 3))
print("\nAttention scores (SHUFFLED computation order):\n", np.round(scores_shuffled, 3))
print("\nIdentical regardless of computation order?", np.allclose(scores_normal, scores_shuffled))

Expected Output:

RNN hidden states (each REQUIRES the previous one):
 [[ 0.943  0.38   0.753  0.978]
 [ 0.903 -0.836  0.812 -0.246]
 [-0.728  0.897  0.406  0.881]
 [ 0.955 -0.64   0.192  0.488]
 [ 0.278  0.056  0.47  -0.756]]




Attention scores (normal order):
 [[ 9.252  3.494  3.382  2.573  0.946]
 [ 3.494  5.368 -0.677  1.674  3.417]
 [ 3.382 -0.677  2.315  0.521 -1.435]
 [ 2.573  1.674  0.521  0.902  0.966]
 [ 0.946  3.417 -1.435  0.966  3.102]]




Attention scores (SHUFFLED computation order):
 [[ 9.252  3.494  3.382  2.573  0.946]
 [ 3.494  5.368 -0.677  1.674  3.417]
 [ 3.382 -0.677  2.315  0.521 -1.435]
 [ 2.573  1.674  0.521  0.902  0.966]
 [ 0.946  3.417 -1.435  0.966  3.102]]




Identical regardless of computation order? True

The RNN function had to be written as a for loop, with each iteration depending on the last — there’s no way to rewrite rnn_forward to compute step 3 before step 2 and get the same result, because step 3 needs step 2’s output as an input.

The attention scores, by contrast, come out byte-for-byte identical (True) whether computed in normal order or a deliberately shuffled order — proof that no position’s computation depends on another position’s result, only on the raw input data. This is the literal, structural reason attention can be computed in parallel on a GPU across an entire sequence, and recurrence fundamentally cannot.

10. How Is This Used in Modern AI?

Where this concept lives

Follow the concept at three levels: inside the model, where the computation happens; inside the AI product, where that computation supports a visible feature; and inside production, where engineers measure speed, memory, quality, and failure cases. The details below connect those levels.

🤖 How Is This Used in Modern AI?

This parallelizability is not a minor implementation convenience — it’s arguably the single biggest practical reason Transformers scaled to the size of modern LLMs at all. Training a model with hundreds of billions of parameters on trillions of tokens requires using GPU/TPU hardware’s massive parallelism efficiently; an architecture that forces step-by- step sequential computation simply cannot use that hardware anywhere near as effectively.

11. How Is This Used in LLMs?

Trace one model call

User text → tokens → Transformer computation → output-token probabilities
          this topic affects one part of that computation

An LLM does not apply this idea as a separate magic step. It uses it as part of the repeated numerical pipeline that transforms token vectors and produces the next-token probabilities.

Every modern LLM — GPT-style, Claude, and virtually every other general-purpose language model — is a decoder-only Transformer (Module 13 covers exactly why decoder-only specifically). None of them use recurrent connections. The entire remainder of this course builds up, piece by piece, exactly what replaced the RNN’s recurrence.

Real systems you can recognize

Google’s original Attention Is All You Need report introduced the Transformer without recurrence and reported stronger translation results with greater training parallelism. GPT-style and Gemini models belong to this Transformer family, although their exact modern architectures differ from the 2017 design.

12. How Is This Used in Agentic AI?

Trace one agent step

Goal + history + tool results
              ↓
     LLM processes the context
              ↓
Suggested answer or tool call
              ↓
Agent runtime validates and executes it

This distinction matters: the Transformer helps produce the proposal, while the surrounding agent software controls tools, permissions, retries, memory, and execution.

Direct relevance to Agentic AI: High, indirectly. Every LLM powering an agent’s reasoning is only practically trainable and servable at today’s scale because of the architectural shift this module explains. Without solving the parallelization problem, the large, capable models that make modern agents possible simply wouldn’t exist in their current form.

When this knowledge is useful

Use Why Transformers? when you need to explain, implement, debug, evaluate, or optimize the corresponding part of a Transformer pipeline. It is also useful when a model API behaves unexpectedly and you need to trace the behavior back to tokens, tensor shapes, attention visibility, training, or inference mechanics.

When it is not enough

Understanding this mechanism does not by itself prove that a complete model or application is accurate, safe, fast, or cost-effective. Production decisions still require representative evaluation data, latency and memory measurements, model-specific documentation, and tests of the surrounding retrieval or agent code.

13. Common Beginner Mistakes

⚠️ Mistake

Incorrect idea: treating “vanishing gradients” and “poor parallelization” as the same problem.

Why it is incorrect: They are genuinely distinct, as Section 5’s table makes explicit. LSTM/GRU addressed the first; only attention addressed the second.

⚠️ Mistake

Incorrect idea: assuming Transformers were adopted purely because they’re “more accurate.”

Why it is incorrect: The parallelization advantage was, historically, at least as decisive as any accuracy improvement — it’s what made training at LLM scale feasible in the first place.

⚠️ Mistake

Incorrect idea: treating this progression as purely historical trivia.

Why it is incorrect: Each transition (Section 6) was a response to a specific, identifiable engineering limitation — understanding why is what makes the rest of this course’s architecture choices feel motivated rather than arbitrary.

14. Important Distinctions

Vanishing Gradients	Poor Parallelization
A gradient-magnitude problem across time steps (DL Module 10, 14)	A computational/structural problem — steps can’t be computed simultaneously
Partially fixed by LSTM/GRU gating	NOT fixed by gating — inherent to any recurrent architecture
Fixed by attention’s direct, distance-independent connections	Fixed by attention’s lack of step-to-step dependency

RNN Hidden State Computation	Attention Score Computation
`h[t]` requires `h[t-1]` — a forced order	Score between any `i`, `j` needs only the raw inputs
Cannot be reordered or parallelized	Provably order-independent (verified above)

15. Production / Engineering Considerations

Training and serving cost at LLM scale would be economically infeasible with a fundamentally sequential architecture — this is a genuine, practical constraint, not an academic footnote.
Understanding why Transformers parallelize well is directly useful when reasoning about inference cost and latency later in this course (Modules 16-17) — the same underlying property (no forced step-by-step dependency within a single forward pass) is relevant there too.

16. Interview Questions

Beginner

Q: What were the two main limitations of RNNs that motivated the shift toward attention?

Ans: Vanishing gradients over long sequences (gradients shrink exponentially as they propagate backward through many time steps, DL Module 14), and an inherently sequential computation pattern — each time step depends on the previous one’s result, which prevents parallelizing computation across a sequence.

Intermediate

Q: Why didn’t LSTM and GRU fully solve the problems RNNs had?

Ans: LSTM/GRU’s gating mechanisms meaningfully reduce vanishing gradients, allowing longer effective dependencies than plain RNNs. But they don’t address the second, structural problem at all — every time step in an LSTM or GRU still requires the previous step’s hidden state to be computed first, making them just as fundamentally sequential and unparallelizable as plain RNNs.

Advanced

Q: Explain, structurally, why attention can be computed in parallel while recurrent architectures cannot.

Ans: A recurrent architecture’s hidden state at time t is defined as a function of the input at t and the hidden state at t-1 — by definition, this creates a hard dependency chain where step t cannot begin until step t-1 completes. Attention’s core computation — the similarity score between any two positions — depends only on those positions’ original input representations, not on any other position’s computed result.

This means every pairwise score in a sequence can be computed independently and simultaneously, which is exactly what was demonstrated: computing attention scores in a shuffled order produced identical results to computing them sequentially, proving no result depends on another result being computed first.

Scenario

Q: A team wants to train a sequence model on a very large dataset and is deciding between an LSTM-based and a Transformer-based architecture. Beyond potential accuracy differences, what practical factor should weigh heavily in this decision?

Ans: Training throughput and hardware utilization. An LSTM’s sequential dependency prevents it from fully utilizing GPU parallelism within a single sequence’s processing, while a Transformer’s attention mechanism can process an entire sequence’s relationships in parallel.

At large dataset and model scale, this difference in hardware utilization translates directly into training time and cost — often the deciding factor in practice, independent of any accuracy considerations.

Architecture

Q: Is it accurate to say Transformers are “more powerful” than RNNs in every sense?

Ans: Not precisely — the clearest, most decisive advantage is parallelizability and better handling of long-range dependencies (via direct, distance-independent connections between any two positions).

Whether this makes a Transformer “more powerful” depends on context; what’s unambiguous is that the architectural shift solved specific, identifiable engineering bottlenecks that were genuinely blocking further scaling of recurrent models.

AI Engineering

Q: Why does understanding this history matter for someone who will never train a Transformer from scratch, only use pretrained LLMs via API?

Ans: Because it explains why the model you’re calling is built the way it is — decoder-only, attention-based, trained on massive parallel hardware — which directly informs your intuition about its behavior: why it can relate distant parts of a long context (attention’s direct connections), why very long contexts are computationally expensive (Module 17), and why the underlying architecture hasn’t meaningfully changed even as models have scaled dramatically.

17. What You Should Remember

RNNs had two distinct problems: vanishing gradients (partially fixed by LSTM/GRU) and sequential computation preventing parallelization (not fixed by LSTM/GRU at all).
Attention solves both simultaneously — verified directly: RNN hidden states are provably order-dependent; attention scores are provably order-independent.
This parallelizability is arguably the single most decisive practical reason Transformers scaled to the size of modern LLMs.

18. How This Helps Me Build AI Systems

Every architectural choice in the rest of this course — attention, multi-head attention, the full Transformer block — traces back to this one structural insight: relating tokens without forcing sequential computation. Understanding this now means every subsequent module will feel like a natural elaboration on a well-motivated idea, not an arbitrary design.

Next: Module 2 — Transformer Input: From Text to Vectors — the complete path from raw text to what actually enters a Transformer.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026