Backward Pass & Backpropagation Through Time

Recap: Forward Equations

$h_t = \tanh(W_{xh} \cdot x_t + W_{hh} \cdot h_{t-1} + b_h)$ $y_t = W_{hy} \cdot h_t + b_y$ $p_t = \text{softmax}(y_t)$ $L_t = -\log(p_t[\text{target}_t])$

Step 1: Gradient of Loss w.r.t. Output (Softmax + Cross-Entropy)

The combined gradient of cross-entropy loss through softmax has a beautifully simple form:

$\frac{\partial L_t}{\partial y_t} = p_t - \mathbf{1}_{\text{target}}$

where $\mathbf{1}_{\text{target}}$ is a one-hot vector with 1 at the target index.

Example: If the target is index 2 and $p_t = [0.1, 0.2, 0.6, 0.1]$:

$dy_t = [0.1, 0.2, 0.6 - 1.0, 0.1] = [0.1, 0.2, -0.4, 0.1]$

The negative value at index 2 says: “increase this logit to make the correct answer more likely.”

Step 2: Output Layer Gradients

Since $y_t = W_{hy} \cdot h_t + b_y$, applying the chain rule:

$\frac{\partial L_t}{\partial W_{hy}} = dy_t \cdot h_t^T, \quad \frac{\partial L_t}{\partial b_y} = dy_t$

Step 3: Gradient Flowing into the Hidden State

The gradient reaches $h_t$ from the output layer: $dh_t = W_{hy}^T \cdot dy_t$.

But $h_t$ affects the loss in two ways — directly (current output) and indirectly (all future outputs). We add the gradients from each path:

$dh_t^{\text{total}} = \underbrace{W_{hy}^T \cdot dy_t}_{\text{from current output}} + \underbrace{dh_{\text{next}}}_{\text{from future timesteps}}$

Step 4: Backpropagating Through tanh

The derivative of tanh is $(1 - h_t^2)$. The symbol $\odot$ denotes element-wise multiplication:

$dz_t = (1 - h_t^2) \odot dh_t$

Step 5: Weight Gradients for the Hidden Layer

$\frac{\partial L_t}{\partial W_{xh}} = dz_t \cdot x_t^T, \quad \frac{\partial L_t}{\partial W_{hh}} = dz_t \cdot h_{t-1}^T, \quad \frac{\partial L_t}{\partial b_h} = dz_t$

Critical: These are accumulated with += across all timesteps because the same weights are shared.

Step 6: Pass Gradient to Previous Timestep

$dh_{\text{next}} = W_{hh}^T \cdot dz_t$

This becomes the $dh_{\text{next}}$ term in Step 3 for the previous timestep, completing the loop.

Gradient Flow Diagram

        L₀          L₁          L₂        (losses)
        ↑           ↑           ↑
       dy₀         dy₁         dy₂        (Step 1: softmax gradient)
        ↑           ↑           ↑
  dWhy,dby    dWhy,dby    dWhy,dby        (Step 2: output weights — accumulated)
        ↑           ↑           ↑
       dh₀    ←    dh₁    ←    dh₂        (Step 3: hidden gradient — two sources)
        ↑           ↑           ↑
     dh_raw₀     dh_raw₁     dh_raw₂     (Step 4: through tanh)
        ↑           ↑           ↑
  dWxh,dWhh,dbh  (all accumulated)         (Step 5: hidden weights)
        ↑           ↑
    dh_next₀    dh_next₁                  (Step 6: backward chain)

The arrows pointing left (←) show the “through time” gradient chain.

If the largest eigenvalue of $W_{hh}$ is $\lambda > 1$:

$\|dh_1\| \approx \lambda^T \cdot \|dh_T\|$

Norm clipping formula:

$g_{\text{clipped}} = \begin{cases} g & \text{if } \|g\| \leq \text{max\_norm} \\ g \cdot \frac{\text{max\_norm}}{\|g\|} & \text{otherwise} \end{cases}$

An alternative is value clipping (clamp each element to [-5, 5]) — simpler but doesn’t preserve gradient direction.

Backprop bugs are notoriously hard to spot — the code runs, loss decreases (slowly), but gradients are subtly wrong. Always verify with finite differences:

$\frac{\partial L}{\partial w} \approx \frac{L(w + \epsilon) - L(w - \epsilon)}{2\epsilon}$

Compare your analytical gradient against this numerical approximation:

Common Mistakes

Dimensional Analysis Cheat Sheet

A powerful debugging technique — if the matrix shapes don’t match, the gradient is wrong:

Where $H$ = hidden size, $V$ = vocabulary size.

Rule of thumb: The gradient of a weight matrix $W$ used as $W \cdot a$ is the outer product of the upstream gradient and $a$.

def backward_pass(targets, cache):
    """Compute gradients via Backpropagation Through Time.

    Note: Wxh, Whh, Why, bh, by are module-level globals
    (the same weight matrices used in forward_pass).
    """
    # Unpack the tuple returned by forward_pass: (xs, hs, ys, ps)
    xs, hs, _ys, ps = cache
    T = len(targets)

    dWxh, dWhh, dWhy = np.zeros_like(Wxh), np.zeros_like(Whh), np.zeros_like(Why)
    dbh, dby = np.zeros_like(bh), np.zeros_like(by)
    dh_next = np.zeros_like(hs[0])

    for t in reversed(range(T)):
        dy = ps[t].copy()
        dy[targets[t]] -= 1                          # Step 1

        dWhy += np.outer(dy, hs[t]); dby += dy       # Step 2
        dh = Why.T @ dy + dh_next                     # Step 3
        dh_raw = (1 - hs[t] ** 2) * dh                # Step 4

        h_prev = hs[t-1] if t > 0 else np.zeros_like(hs[0])
        dWxh += np.outer(dh_raw, xs[t])               # Step 5
        dWhh += np.outer(dh_raw, h_prev)
        dbh  += dh_raw

        dh_next = Whh.T @ dh_raw                      # Step 6

    return dWxh, dWhh, dWhy, dbh, dby

def clip_gradients(grads, max_norm=5.0):
    """Scale gradients if their total norm exceeds max_norm."""
    total_norm = np.sqrt(sum(np.sum(g ** 2) for g in grads))
    if total_norm > max_norm:
        scale = max_norm / total_norm
        return [g * scale for g in grads]
    return grads

def numerical_gradient(param, idx, loss_fn, eps=1e-5):
    """Verify analytical gradients with finite differences."""
    old_val = param.flat[idx]
    param.flat[idx] = old_val + eps; loss_plus = loss_fn()
    param.flat[idx] = old_val - eps; loss_minus = loss_fn()
    param.flat[idx] = old_val
    return (loss_plus - loss_minus) / (2 * eps)

For sequence length $T$, hidden size $H$, vocabulary size $V$:

BPTT has the same time complexity as the forward pass. Memory scales linearly with sequence length because we store all hidden states for backprop.

Because $dW_{hh,0} = dz_0 \cdot h_{-1}^T = dz_0 \cdot \mathbf{0}^T = \mathbf{0}$.

4 times — one multiplication by $W_{hh}^T$ per step backward. This is why gradients vanish/explode over long sequences.

What we skipped on this page:

• Backpropagation. Rumelhart, Hinton & Williams, “Learning Representations by Back-propagating Errors” (1986) established backpropagation as the standard training algorithm for neural networks. Every gradient computation on this page is an application of the chain rule formalized in that paper.

• Backpropagation Through Time. Werbos, “Backpropagation Through Time: What It Does and How to Do It” (1990) extended backpropagation to recurrent networks by unrolling the RNN across timesteps and applying the standard chain rule to the unrolled graph. The six-step procedure on this page is BPTT.

• Gradient clipping and the exploding gradient problem. Pascanu, Mikolov & Bengio, “On the Difficulty of Training Recurrent Neural Networks” (2013) analyzed why RNN gradients explode or vanish, connecting the problem to the eigenvalues of $W_{hh}$. They proposed gradient norm clipping as the practical fix for explosions (vanishing requires architectural changes, covered in Vanishing Gradients & Gated RNNs).

• Truncated BPTT in practice. Williams & Peng, “An Efficient Gradient-Based Algorithm for On-Line Training of Recurrent Network Trajectories” (1990) introduced truncated BPTT, where gradients flow backward through a fixed window rather than the entire sequence. This trades gradient accuracy for memory savings and is standard in all modern RNN training.