Vanishing Gradients & Gated RNNs

In the backward pass, we saw how gradients flow through time via repeated multiplication by $W_{hh}^T$. Now we’ll see why this causes a fundamental problem — and the elegant architectural solutions that fix it.

The Problem

Train an RNN on long sequences and plot gradient magnitude at each timestep:

What to look for: The bars represent gradient magnitude at each timestep. Notice how bars on the left (early tokens) are dramatically smaller than the right (near the loss). The gradient doesn’t just decrease, it drops exponentially. By t=0, almost no learning signal remains.

Try it yourself: Drag the |λ| slider below 0.9 to see gradients vanish, or above 1.2 to see them explode. Then increase the sequence length T to see how the problem gets worse with longer sequences.

Consequence: early tokens don’t learn. The network can’t capture long-range dependencies. A word at the start of a sentence has almost no influence on the loss at the end.

Why Gradients Vanish

The Core Intuition

Each backward step through time multiplies the gradient by a factor. Think of it like a game of telephone: the message (gradient) gets distorted at every step. If each step slightly shrinks the signal, after enough steps the message is gone entirely.

When you repeatedly multiply a number by a constant, the result grows or shrinks exponentially. Multiplying by 0.9 a hundred times: $0.9^{100} \approx 0.00003$. Multiplying by 1.1 a hundred times: $1.1^{100} \approx 13{,}780$.

The same thing happens with weight matrices in an RNN. The “multiplication factor” of a matrix is determined by its largest eigenvalue (how much the matrix stretches vectors). The key intuition is: repeated matrix multiplication either shrinks or grows signals exponentially, and for typical randomly-initialized weight matrices, shrinking (vanishing) is the default.

On top of that, the tanh activation further shrinks gradients at every step. When hidden state values are large (near -1 or 1), tanh is nearly flat, so its derivative is near zero. These two effects compound: even if the weight matrix factor is near 1, saturated tanh neurons still kill the gradient.

Why This Matters

Consider: “The cat, which was sitting on the mat in the corner of the room, was sleeping.”

The subject “cat” determines “was” (not “were”), but they’re 14 words apart. With vanishing gradients, the signal from “was” barely reaches “cat” — the model can’t learn this dependency.

Practical Limits of Vanilla RNNs

Attempted Fixes (Partial)

The real fix requires a change in architecture — replacing multiplicative state updates with additive ones.

The Solution: LSTM

The Key Insight: Addition vs Multiplication

In vanilla RNN, the hidden state is overwritten at each step via matrix multiplication. Gradients flow through tanh and decay. What if instead of overwriting, we added to the state?

LSTM introduces a cell state, a separate memory track updated via addition instead of multiplication. Think of it like a conveyor belt running above the main processing: information can be added to the belt or removed from it, but by default it just passes through unchanged. The $+$ creates a gradient highway: if the forget gate is near 1, gradients pass through unchanged.

Step through one timestep to see how it works:

What to look for: Watch how the cell state (top row) flows through. The forget gate scales the old memory, then new information gets added on top. That + sign is the whole trick. In a vanilla RNN, information passes through matrix multiply and tanh at every step (shrinking). Here, it passes through a simple addition (preserving).

The Four Gates

Each gate looks at the previous hidden state and the current input, then outputs a value between 0 and 1 (via sigmoid): 0 means “block everything,” 1 means “let everything pass.”

Worked Example

Walk through one timestep with the values from the visualization above. Previous cell state is $[0.80, 0.50]$:

Step 1: Forget gate decides what to keep. $f_t = [0.90, 0.10]$. Keep 90% of dimension 1, discard 90% of dimension 2. $f_t \odot c_{t-1} = [0.90 \times 0.80,\ 0.10 \times 0.50] = [0.72,\ 0.05]$

Step 2: Input gate + candidate decide what to write. $i_t = [0.30, 0.80]$ and $\tilde{c}_t = [-0.60, 0.70]$. $i_t \odot \tilde{c}_t = [0.30 \times (-0.60),\ 0.80 \times 0.70] = [-0.18,\ 0.56]$

Step 3: Cell state update via addition. $c_t = [0.72,\ 0.05] + [-0.18,\ 0.56] = [0.54,\ 0.61]$

Step 4: Output gate controls what to expose. $o_t = [0.70, 0.50]$. $h_t = o_t \odot \tanh(c_t) = [0.70 \times 0.49,\ 0.50 \times 0.54] = [0.35,\ 0.27]$

The cell state updated through addition, not through a weight matrix and tanh. That’s why gradients survive.

Why LSTM Prevents Vanishing Gradients

Because the cell state is updated by adding (not multiplying through tanh), the gradient flows straight back through time along the cell state highway. The only thing that can weaken it is the forget gate, which is typically learned to be close to 1 for important information.

Compare the two architectures over 100 timesteps:

The forget gate is a learned value near 1, not a full matrix multiplication through tanh. This is the fundamental difference.

GRU: A Simpler Alternative

GRU achieves similar results with only 2 gates and 1 state (no separate cell state):

How GRU Works

GRU has two gates:

• Reset gate ($r_t$) — controls how much of the previous hidden state to use when computing the new candidate. Setting it to 0 lets the model “forget” the past and start fresh.
• Update gate ($z_t$) — controls the balance between keeping the old state and adopting the new candidate. It replaces LSTM’s separate forget and input gates — they’re tied together since keeping old info ($1 - z_t$) and writing new info ($z_t$) must sum to 1.

The update gate $z_t$ works like a slider: $z_t = 0$ means “keep the old state entirely” (maximum memory), while $z_t = 1$ means “replace with the new candidate entirely.”

LSTM vs GRU

class LSTMCell:
    def __init__(self, input_size, hidden_size):
        self.hidden_size = hidden_size
        scale = 0.1
        # Combined weights: [W_f, W_i, W_c, W_o] stacked for efficiency
        self.W = np.random.randn(4 * hidden_size, input_size + hidden_size) * scale
        self.b = np.zeros(4 * hidden_size)
        self.b[:hidden_size] = 1.0  # Forget gate bias = 1 (remember by default)
    
    def forward(self, x_t, h_prev, c_prev):
        H = self.hidden_size
        concat = np.concatenate([h_prev, x_t])
        gates = self.W @ concat + self.b  # All gates in one matmul
        
        f_t = sigmoid(gates[0*H:1*H])      # Forget
        i_t = sigmoid(gates[1*H:2*H])      # Input
        c_tilde = np.tanh(gates[2*H:3*H])  # Candidate
        o_t = sigmoid(gates[3*H:4*H])      # Output
        
        c_t = f_t * c_prev + i_t * c_tilde  # Cell state (ADDITION!)
        h_t = o_t * np.tanh(c_t)
        return h_t, c_t

Connection to Transformers

You’ll learn about self-attention in detail in Self-Attention. For now, here’s a preview of why Transformers solve the gradient problem.

In self-attention (covered later), every token can directly “look at” every other token — no matter how far apart they are. The gradient from any token to any other token passes through a single multiplication, not a chain of multiplications that grows with distance. This is the key difference:

This is a fundamental reason why Transformers replaced RNNs for most tasks.

When to Use What

Need to model sequences?
├── Sequence length < 50? → Vanilla RNN might work
├── Can parallelize?
│   ├── Yes → Transformer (preferred)
│   └── No → LSTM/GRU
└── Training speed critical?
    ├── Yes → GRU
    └── No → LSTM

Modern Perspective

Since 2017, Transformers have largely replaced LSTMs for most NLP tasks. LSTMs remain useful for:

• Streaming/online processing (one token at a time)
• Memory-constrained deployments
• Time series with strict ordering

Quick Summary

Next Steps

Before building full transformer blocks, there’s one more building block: normalization and skip connections. These techniques stabilize training and appear in every modern architecture. Next: Layer Norm & Residuals →