RNN Forward Pass

The hidden state update equation:

$h_t = \tanh\!\left(W_{xh} \cdot x_t + W_{hh} \cdot h_{t-1} + b_h\right)$

The subscripts tell you what each weight matrix connects:

• $W_{xh}$ = weights from x (input) to h (hidden) — shape: (hidden_size, vocab_size)
• $W_{hh}$ = weights from h (hidden) to h (hidden) — shape: (hidden_size, hidden_size)
• $b_h$ = bias for the hidden layer — shape: (hidden_size,)

Why two separate weight matrices? One processes “what am I seeing now?” ($W_{xh} \cdot x_t$) and the other processes “what do I remember?” ($W_{hh} \cdot h_{t-1}$). Their results are added together — the network learns how to balance new input vs. memory.

Logits from a linear projection of the hidden state:

$y_t = W_{hy} \cdot h_t + b_y$

Softmax converts logits to probabilities:

$p_i = \frac{e^{y_i}}{\sum_j e^{y_j}}$

Why exponential, not just dividing by the sum?

1. $e^x$ is always positive — no negative probabilities
2. $e^x$ grows rapidly — amplifies small differences in logits
3. Dividing by the total ensures everything sums to 1

Cross-entropy loss at timestep $t$:

$L_t = -\log\!\left(p_t[\text{target}]\right)$

Why negative log?

• When $p \approx 1$ (correct and confident): $-\log(1) = 0$ — zero loss, perfect
• When $p \approx 0$ (wrong): $-\log(0.001) = 6.9$ — huge penalty
• The log creates an asymmetric punishment: being confidently wrong is punished exponentially more

The total loss sums across all timesteps:

$L = \sum_t L_t = -\sum_t \log\!\left(p_t[\text{target}_t]\right)$

Here’s the full algorithm. Notice the numerically stable softmax (subtract max before exponentiation):

def forward_pass(inputs, targets, h_prev):
    """
    Process a sequence and compute loss.

    Args:
        inputs:  List of character indices [t0, t1, ...]
        targets: List of target indices, shifted by 1
        h_prev:  Initial hidden state (hidden_size x 1)

    Returns:
        loss:   Total cross-entropy loss
        cache:  Intermediate values needed for backprop
    """
    xs, hs, ys, ps = {}, {}, {}, {}
    hs[-1] = h_prev
    loss = 0

    for t in range(len(inputs)):
        # Step 1: One-hot encode input
        xs[t] = np.zeros((vocab_size, 1))
        xs[t][inputs[t]] = 1

        # Step 2: Update hidden state
        hs[t] = np.tanh(Wxh @ xs[t] + Whh @ hs[t-1] + bh)

        # Step 3: Compute output logits
        ys[t] = Why @ hs[t] + by

        # Step 4: Numerically stable softmax
        shifted = ys[t] - np.max(ys[t])
        ps[t] = np.exp(shifted) / np.sum(np.exp(shifted))

        # Step 5: Accumulate cross-entropy loss
        loss += -np.log(ps[t][targets[t], 0] + 1e-8)

    return loss, (xs, hs, ys, ps)

Numerical Stability in Softmax

The raw softmax formula can silently overflow when logits are large (e.g., $e^{1000} = \infty$). The fix is to subtract the max before exponentiation:

shifted = y - np.max(y)          # safe: largest value becomes 0
p = np.exp(shifted) / np.sum(np.exp(shifted))

Subtracting the max doesn’t change the final probabilities (the constant cancels in the division) but prevents floating-point overflow.

Why Cross-Entropy?

Cross-entropy has three properties that make it ideal for classification:

• Simple gradient: $\frac{\partial L}{\partial y_i} = p_i - \mathbf{1}_{i=\text{target}}$ — just subtract 1 from the target’s probability
• Punishes confident wrong answers: Being 99% sure of the wrong answer is penalised far more than being 51% sure
• Probabilistic grounding: Minimising cross-entropy is equivalent to maximising the likelihood of the training data

Truncated Backpropagation Through Time

For very long sequences (an entire book), backpropagating through every single timestep is impractical. Instead:

1. Process a chunk (e.g., 100 characters)
2. Compute gradients for that chunk only
3. Move to the next chunk, carrying the hidden state forward

The hidden state preserves context across chunks, but gradients only flow within each chunk. This trades a small amount of gradient accuracy for a dramatic reduction in memory.

Temperature Sampling

When generating text (not training), we can control creativity with temperature $\tau$:

$p_i = \frac{e^{y_i / \tau}}{\sum_j e^{y_j / \tau}}$

• $\tau < 1$: Sharpens the distribution — more confident, less varied output
• $\tau > 1$: Flattens the distribution — more random, more creative output
• $\tau \to 0$: Always picks the single highest-probability token (greedy decoding)

Key References

• The original RNN forward pass. Elman, “Finding Structure in Time” (1990) defined the simple recurrent network: feed the previous hidden state back as input alongside the current token. The hidden state update equation on this page ($h_t = \tanh(W_{xh} \cdot x_t + W_{hh} \cdot h_{t-1} + b_h)$) is exactly the Elman network.

• min-char-rnn. Karpathy, “The Unreasonable Effectiveness of Recurrent Neural Networks” (2015) walked through a minimal character-level RNN in ~100 lines of Python. The implementation on this page follows the same structure. The blog post remains one of the best introductions to RNNs in practice.

• Softmax origins. Bridle, “Training Stochastic Model Recognition Algorithms as Networks can Lead to Maximum Mutual Information Estimation of Parameters” (1989) introduced softmax as the output activation for neural network classifiers, replacing the ad-hoc normalizations used before.