RNNs, LSTM and GRU | TechByteByByte

Begin with the central question

How can a network carry information from earlier words into a later decision?

That question is the reason this topic exists. Keep it in mind as each new term appears: every equation, diagram, and code example below is one part of the answer.

current input + previous state → new state → next step

Before you continue: three tools for this module

Sequence: ordered data such as words, audio samples, or daily measurements.
Hidden state: a running numeric summary passed from one sequence step to the next.
Gate: a learned control that decides how much information to keep, add, or forget.

You do not need to memorize these definitions yet. Use them as a small map whenever the terms appear below.

What You Will Understand

Why sequence data (like text) needs different handling than the fixed-size inputs Modules 2-13 assumed, how RNNs process sequences using a recurrent hidden state, why they suffer especially badly from vanishing gradients, how LSTM/GRU gates helped — and precisely why these architectures’ remaining limitations directly motivated the attention mechanism (Module 15).

An RNN updates a hidden state one step at a time:

x₁ → h₁ → h₂ → h₃ → ...
           ↑     ↑
          x₂    x₃

LSTM/GRU gates control what is retained, added, and exposed

Gates improve gradient flow and memory, but they do not preserve every detail perfectly. Sequential processing also limits parallelism, which helped motivate attention-based architectures.

Why Ordered, Variable-Length Data Needs Memory

Every network so far processes one fixed-size input and produces one output. Text is different: a sentence has variable length, and the order of words matters — “dog bites man” and “man bites dog” contain the same words but mean something entirely different. RNNs exist to process inputs of variable length, one element at a time, while maintaining some notion of what’s come before.

Reading While Updating a Running Summary

an RNN reads a sequence one token at a time, like a person reading a sentence word by word, maintaining a running “mental summary” (the hidden state) that gets updated after each new word. This summary is meant to carry forward everything relevant from earlier in the sequence — but, as you’ll see demonstrated concretely, that summary tends to lose earlier information the further it has to travel.

Analogy: The Reading Journal vs. The Filing Cabinet with Gates Imagine reading a long mystery book and tracking details to solve the crime:

Standard RNN (The Small Reading Journal): As you read each page, you write a brief summary in a tiny, pocket-sized notepad (the hidden state). Because the notepad is so small, you must overwrite old summaries to make room for new ones. By Chapter 15, you have completely overwritten details from Chapter 1 (vanishing hidden state memory / short-term memory bottleneck).

LSTM (The Filing Cabinet with Gates): You are equipped with a filing cabinet containing three highly organized desk clerks (gates) managing the file folders (the cell state $C_t$ ):

Forget Gate: A clerk inspects the folder and discards irrelevant information. (e.g. “The suspect changed rooms, throw away the old room number”).

Input Gate: A clerk decides which new clues from the current page are important enough to write down and add to the folder. (e.g. “Write down that the suspect bought a knife”).

Output Gate: A clerk decides which files in the folder should be extracted right now to update your pocket-notebook (the current hidden state $h_t$ ) to make immediate decisions.

The cell state behaves like a dedicated file cart rolling down a straight corridor. Because the gates selectively edit it rather than forcing it to undergo full matrix multiplications at every single step, the memory remains pristine and intact over hundreds of pages.

📊 Visual Flowchart: LSTM Gating Architecture

Here is how the gates regulate information flow into and out of the cell state ( $C_t$ ):

graph TD
    PrevC["Previous Cell State (C_t-1)"] --> CellHighway["Cell State Highway (C_t)"]
    CellHighway --> NextC["Next Cell State (C_t)"]

PrevH["Previous Hidden State (h_t-1)"] --> GateInputs["Gate Inputs"]
    CurrX["Current Input (x_t)"] --> GateInputs

subgraph LSTMCell ["LSTM Cell Internals"]
        GateInputs --> ForgetGate["1. Forget Gate (f_t)<br>f_t = σ(Wf * [h_t-1, x_t] + bf)"]
        GateInputs --> InputGate["2. Input Gate (i_t)<br>i_t = σ(Wi * [h_t-1, x_t] + bi)"]
        GateInputs --> CandidateState["3. Candidate State (~C_t)<br>~C_t = tanh(Wc * [h_t-1, x_t] + bc)"]
        GateInputs --> OutputGate["4. Output Gate (o_t)<br>o_t = σ(Wo * [h_t-1, x_t] + bo)"]

ForgetGate -->|Multiply to discard| CellHighway
        InputGate -->|Multiply to scale updates| UpdateNode["Add updates (+ i_t * ~C_t)"]
        CandidateState --> UpdateNode
        UpdateNode --> CellHighway

CellHighway --> FinalTanh["tanh(C_t)"]
        FinalTanh -->|Multiply by o_t| OutNode["Multiply Node"]
        OutputGate --> OutNode
    end

OutNode --> NextH["Next Hidden State (h_t)"]

4. Core Concept

Term	Definition
Sequence	An ordered list of inputs (e.g., words in a sentence)
Hidden state	The RNN’s running “memory,” updated at every step, carrying information forward
Recurrent connection	The hidden state feeds back into the network at the next time step — the defining feature of an RNN
Vanishing gradient (through time)	Gradients shrinking as they propagate backward across many time steps, exactly as Module 10 showed across layers

The RNN update, precisely

hidden_state[t] = tanh(Wx @ input[t] + Wh @ hidden_state[t-1] + b)

Every time step reuses the same Wx, Wh, b — exactly like a CNN’s filter (Module 13) being reused across spatial positions, an RNN reuses the same weights across time positions.

5. How It Works — Step by Step

1. Initialize the hidden state (commonly all zeros)
2. FOR each element in the sequence, IN ORDER:
   a. Combine the current input with the PREVIOUS hidden state
      (using the SAME weights every step)
   b. Apply an activation function (commonly tanh)
   c. This becomes the NEW hidden state
3. The final hidden state (after processing the whole sequence)
   is a "summary" that can be used for a prediction -- or the
   hidden state at EVERY step can be used, depending on the task

6. Mathematical Intuition

First, use only small numbers

At step 1 the state might be 0. Reading a positive clue adds 0.8; reading an irrelevant word keeps most of it; reading a contradiction may reduce it. Real RNN states are vectors and their updates are learned, but the idea is still controlled memory over time.

Read the mathematics as a story

An RNN repeatedly updates a hidden state. LSTM and GRU gates control what to remember, write, and forget, helping important signals survive longer sequences.

current input + previous state → new state → next step

Do not begin by memorizing the symbols. First identify what enters, what operation changes it, and what comes out. The symbols are a compact description of that journey.

Why RNNs suffer vanishing gradients especially badly: backpropagating through an RNN means applying the chain rule not just across layers (Module 10), but across every time step — and the recurrent weight matrix Wh gets multiplied in at every single step, compounding exactly like Module 10’s sigmoid example, but now over potentially hundreds of time steps for a long sequence, not just a fixed number of layers.

Gradient reaching an early time step
  ≈ (product of ~30 local derivatives, one per time step)

7. Simple Example

Walk through the example

Read the example in three passes:

Identify the input numbers and what each number represents.
Follow one operation at a time instead of jumping directly to the answer.
Interpret the final number in ordinary language and connect it back to the problem.

The purpose is not merely to calculate the result. It is to make the internal mechanism visible. For a short sentence, an RNN’s hidden state after the last word can reasonably still reflect the first word’s influence.

For a long paragraph, the gradient signal connecting the final loss back to the first word has to survive being multiplied by a local derivative at every single intervening word — for long enough sequences, this signal effectively vanishes, and the network can’t learn long-range dependencies (like a pronoun late in a paragraph referring back to a name mentioned many sentences earlier).

8. Python Example

Three Python symbols used below

NumPy (np) is a Python library for working efficiently with lists and grids of numbers.
np.array(...) creates a numeric vector or matrix.
@ performs matrix multiplication: many connected weighted sums calculated together.

You can understand the concept without memorizing the syntax. First follow what the numbers represent, and then connect each code operation to the worked example.

What the code will demonstrate

Before running the code, predict the flow: create a small input, apply the topic’s calculation, and inspect the intermediate or final values. The example uses small numbers so you can connect each printed result to the explanation above; a real model performs the same kind of operation with much larger tensors and learned parameters.

# Build a tiny, inspectable example of RNNs, LSTM and GRU.
# Follow the intermediate values; they reveal what the model is doing.
import numpy as np

def tanh(z): return np.tanh(z)

np.random.seed(7)
sequence = [np.array([0.5, 0.1]), np.array([0.2, 0.8]), np.array([0.9, 0.3]), np.array([0.1, 0.6])]

hidden_size = 3
input_size = 2

Wx = np.random.randn(hidden_size, input_size) * 0.5
Wh = np.random.randn(hidden_size, hidden_size) * 0.5
b = np.zeros(hidden_size)

h = np.zeros(hidden_size)   # initial hidden state

print("Processing sequence step by step:")
for t, x_t in enumerate(sequence):
    h = tanh(Wx @ x_t + Wh @ h + b)
    print(f"  Step {t+1}: input={x_t}, hidden_state={np.round(h, 4)}")

print("\nFinal hidden state (a 'summary' of the whole sequence):", np.round(h, 4))

Expected Output:

Processing sequence step by step:
  Step 1: input=[0.5 0.1], hidden_state=[ 0.3794  0.0286 -0.1946]
  Step 2: input=[0.2 0.8], hidden_state=[-0.1406  0.2803  0.0377]
  Step 3: input=[0.9 0.3], hidden_state=[ 0.4335 -0.0571 -0.4065]
  Step 4: input=[0.1 0.6], hidden_state=[-0.209   0.2975  0.1268]

Final hidden state (a 'summary' of the whole sequence): [-0.209   0.2975  0.1268]

Now, vanishing gradients specifically through time (not layers):

# Build a tiny, inspectable example of RNNs, LSTM and GRU.
# Follow the intermediate values; they reveal what the model is doing.
def tanh_deriv(z): return 1 - np.tanh(z)**2

grad = 1.0
z_values = np.random.randn(30)   # simulate 30 timesteps
for t, z in enumerate(z_values):
    local_grad = tanh_deriv(z) * 0.8   # 0.8 approximates typical recurrent-weight scale
    grad *= local_grad
    if t + 1 in [1, 5, 10, 20, 30]:
        print(f"Gradient magnitude after {t+1} RECURRENT steps: {grad:.10f}")

Expected Output:

Gradient magnitude after 1 RECURRENT steps: 0.1572712219
Gradient magnitude after 5 RECURRENT steps: 0.0075661879
Gradient magnitude after 10 RECURRENT steps: 0.0000191080
Gradient magnitude after 20 RECURRENT steps: 0.0000000000
Gradient magnitude after 30 RECURRENT steps: 0.0000000000

9. How It Works

The hidden state genuinely changes at every step, reflecting the accumulated influence of every input seen so far — notice each step’s hidden state depends on both the current input AND the previous hidden state, exactly the recurrent connection described in Section 4.
The vanishing-gradient-through-time demonstration is essentially Module 10’s exact experiment, but now the “layers” are time steps — and the result is the same catastrophic shrinkage: by 20 steps back, the gradient is already effectively zero. For a 30-word sentence (a perfectly normal length), information from near the start has essentially no ability to influence learning based on something that happened at the end — the concrete, numeric reason plain RNNs struggle with long-range dependencies.

10. Real-World Example

An RNN-based translation system processing a long sentence would tend to translate the last few words accurately (their gradient path back to relevant context is short) while struggling with dependencies spanning the whole sentence — e.g., correctly matching a verb’s tense to a subject mentioned many words earlier. This specific, well-documented weakness is exactly what drove research toward better architectures.

11. LSTM and GRU — Why They Helped

LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit) introduce gates — small learned mechanisms that control how much information to keep, forget, or add to the hidden state at each step, rather than completely overwriting it every time.

Plain RNN:   hidden_state completely recomputed every step
             (nothing explicitly PROTECTED from being overwritten)

LSTM/GRU:    GATES learn to selectively PRESERVE important
             information across many steps, and selectively
             FORGET irrelevant information -- providing a more
             direct path for gradients to flow backward without
             being multiplicatively shrunk at every single step

🧠 This meaningfully reduces (does not eliminate) the vanishing gradient problem, letting LSTM/GRU-based models handle noticeably longer dependencies than plain RNNs — which is precisely why they became the standard sequence architecture for years, before attention (Module 15).

⚠️ Per this course’s scope, gate equations aren’t derived here — the concept (gates provide a more protected path for information and gradients to flow across time) is what matters for an AI engineer; implementation-level gate mechanics are not.

12. RNN/LSTM/GRU Limitations — Motivating Attention

Beyond vanishing gradients (meaningfully reduced but not eliminated by LSTM/GRU), recurrent architectures share a deeper structural limitation:

SEQUENTIAL COMPUTATION:  step 2 cannot be computed until step 1
                          finishes (each step depends on the
                          previous hidden state) -- this makes
                          RNNs/LSTMs/GRUs fundamentally hard to
                          PARALLELIZE across a sequence, even on
                          fast hardware like GPUs

LONG-RANGE DEPENDENCIES:  even with gates, information from very
                          early in a long sequence still has to
                          pass through every intermediate step

🧠 Poor parallelization specifically became an enormous practical problem once training data and model sizes grew to modern scale — a sequential architecture can’t take full advantage of GPUs’ core strength (massive parallelism), directly limiting how large and how fast these models could be trained.

This — not just vanishing gradients — is a central reason the field moved toward attention and Transformers (Module 15-16), which process an entire sequence’s relationships in parallel.

13. How Is This Used in Modern AI?

Follow it from mechanism to product

RNNs, LSTMs, and GRUs remain useful for some streaming, time-series, and resource-constrained tasks. Most current general-purpose LLMs instead use Transformers because attention handles long-range relationships and parallel training more effectively.

How this connects to LLMs

prompt → tokens → deep-learning computations → next-token probabilities → generated response

The model computation is only the middle of the journey. Tokenization happens before it, while decoding and application controls happen afterward; the following example identifies this topic’s exact role.

🤖 Real-world connection

Modern LLMs are not RNN/LSTM/GRU-based — they’re Transformer-based, specifically because Transformers solve both of this module’s core limitations (long-range dependencies and poor parallelization) simultaneously, via attention. RNNs/LSTMs still appear in some specialized settings (certain time-series or streaming applications), but for modern NLP and LLMs specifically, they’ve been largely superseded.

Concept	Where it still appears
Recurrent processing	Some time-series forecasting, certain streaming/low-latency applications
Hidden state	The conceptual ancestor of ideas that persist in modern architectures, though implemented very differently
Vanishing gradients	The same underlying chain-rule mechanism from Module 10, here shown compounding across time instead of layers

14. How Is This Used in Agentic AI?

Trace one agent step

goal + history + tool results → LLM proposal → runtime validation → tool or response

The deep-learning model produces a prediction or structured proposal. The agent runtime—ordinary software around the model—controls permissions, executes tools, stores state, handles retries, and decides whether another model call is needed.

Direct relevance to Agentic AI: Low, directly. Essentially no modern agent system is built on raw RNNs/LSTMs for its core reasoning — that role belongs to Transformer-based LLMs (Modules 15-17).

This module’s real value for an agent-focused engineer is entirely historical/ motivational: understanding precisely why these architectures were replaced is what makes attention and Transformers feel like a natural, well-motivated next step rather than an arbitrary architectural choice.

15. Common Beginner Mistakes / Misconceptions Corrected

⚠️ Mistake

Incorrect idea: LSTM/GRU completely solve vanishing gradients.

Why it is incorrect: They meaningfully reduce the problem via gating, allowing much longer effective dependencies than plain RNNs — but don’t eliminate it entirely, especially for very long sequences.

⚠️ Mistake

Incorrect idea: RNNs are now completely obsolete and never used.

Why it is incorrect: They remain reasonable choices for certain smaller-scale, streaming, or time-series tasks — they’ve specifically been superseded for large-scale language modeling, where Transformers’ parallelization advantage matters enormously.

⚠️ Mistake

Incorrect idea: the main problem with RNNs was just vanishing gradients.

Why it is incorrect: As Section 12 emphasizes, poor parallelization is at least as significant a practical limitation at modern training scale — a distinct issue from vanishing gradients, and arguably the more decisive one in motivating the shift to Transformers.

16. Important Distinctions

RNN	LSTM / GRU
Hidden state fully recomputed every step	Gates selectively preserve/forget information
More prone to vanishing gradients over long sequences	Meaningfully more robust to vanishing gradients, not immune
Fewer parameters, simpler	More parameters, more complex, generally stronger on longer sequences

Vanishing Gradients (this module)	Poor Parallelization (this module)
A gradient-magnitude problem, exactly Module 10’s mechanism, across time	A computational/architectural problem — steps can’t be computed simultaneously
Partially addressed by LSTM/GRU gating	NOT addressed by gating — inherent to any recurrent, step-by-step architecture

17. When to Use

Consider RNN/LSTM/GRU architectures for smaller-scale sequential or streaming tasks (e.g., simple time-series forecasting) where sequence lengths are modest and Transformer-scale infrastructure is unnecessary overhead.

18. When Not to Use

Don’t reach for RNN/LSTM/GRU for modern large-scale language modeling or any task where long-range dependencies and training-time parallelization matter — Transformers (Module 16) are the established, superior choice for exactly these reasons.

19. Interview Questions

Beginner

Q: Why can’t a standard feedforward network directly process a sentence the way an RNN can?

Ans: A feedforward network expects a fixed-size input and has no built-in concept of order or sequence — a sentence has variable length, and word order carries meaning. An RNN processes a sequence one element at a time, maintaining a hidden state that carries forward information from earlier elements, which a plain feedforward network has no mechanism for.

Intermediate

Q: Why do RNNs suffer especially badly from vanishing gradients, compared to a similarly-deep feedforward network?

Ans: Backpropagating through an RNN requires applying the chain rule across every time step in the sequence, not just across a fixed number of layers — a long sequence can have hundreds of time steps, meaning the gradient gets multiplied by a local derivative that many times, exactly compounding the shrinkage demonstrated in this module (and structurally identical to Module 10’s layer-based vanishing gradient problem, just applied along the time dimension instead).

Advanced

Q: LSTM and GRU meaningfully improved long-range dependency handling, yet the field still moved to attention and Transformers. Why weren’t LSTM/GRU sufficient?

Ans: Gating reduces, but doesn’t eliminate, the vanishing gradient problem for very long sequences. More decisively, RNN/LSTM/GRU architectures are fundamentally sequential — each time step’s computation depends on the previous step’s hidden state, which prevents meaningful parallelization across a sequence even on highly parallel hardware like GPUs.

As training data and model sizes grew, this parallelization bottleneck became a severe practical limitation independent of the gradient issue.

Self-attention (Module 15) creates direct paths between positions and allows sequence positions to be processed in parallel during training. This greatly reduces those two limitations, although it introduces its own memory and compute costs.

Scenario

Q: You’re processing very long documents (thousands of words) with an LSTM-based model and notice it performs well on short documents but poorly captures relationships between the beginning and end of long ones. What’s the underlying cause, and what would you consider instead?

Ans: Even with LSTM’s gating, information and gradients connecting very distant positions in a long sequence still have to traverse every intermediate step — for sequences of thousands of words, this remains a genuine, well-documented limitation, not something gating fully resolves.

I’d consider a Transformer-based architecture instead, since self-attention (Module 15) connects any two positions in a sequence directly, regardless of distance, sidestepping this specific limitation entirely.

AI Engineering

Q: Why is it accurate to say modern LLMs are not built on RNNs, LSTMs, or GRUs?

Ans: Modern LLMs are built on the Transformer architecture (Module 16), which processes sequences using self-attention rather than a recurrent hidden state.

This was a deliberate architectural shift, directly motivated by RNN/LSTM/GRU’s two core limitations covered in this module: vulnerability to vanishing gradients over long sequences, and — more decisively at modern scale — an inherently sequential computation pattern that resists the parallelization massive GPU training depends on.

20. What You Should Remember

RNNs process sequences via a recurrent hidden state, reusing the same weights at every time step.
They suffer especially severe vanishing gradients — demonstrated here shrinking to effectively zero by 20 time steps.
LSTM/GRU gating meaningfully reduces, but doesn’t eliminate, this problem.
Beyond gradients, sequential computation prevents parallelization — a distinct, arguably more decisive limitation at modern training scale.
Both limitations together are what directly motivated attention (Module 15).

21. How This Helps Me Build AI Systems

You now understand, with a real numeric demonstration, precisely why the field moved away from recurrent architectures — not as an arbitrary trend, but as a direct, well-motivated response to two specific, measurable problems. This sets up Module 15 to feel like the natural next step it genuinely was, not an unexplained leap.

Next: Module 15 — Attention — the mechanism that substantially reduces both of these module’s core limitations at once, built from first principles with real Q/K/V calculations.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026