Introduction to RNNs

Three-panel comic: 'No Memory' shows a confused robot holding 'BANK' with competing thought bubbles (river vs building); 'Memory' shows a robot on a conveyor belt reading 'I SAT BY THE RIVER BANK' with only the river thought bubble; 'The Loop' shows the core RNN mechanism with new word and old memory feeding into gears to produce new memory and a prediction card

What You’ll Learn

You’ll learn why sequential data needs memory, because the meaning of each word can depend on what came before. You’ll see how an RNN carries a hidden state forward and uses the same loop to combine new input, previous context, and prediction.

The Problem: Why Sequences Are Hard

We now know how to turn text into token IDs (Tokenization), look up each ID as a dense vector (Embeddings), and transform vectors with a simple neural layer (Neural Network Basics). But each token vector is still independent. The embedding for “bank” is the same whether the sentence is about rivers or money. To make predictions, the model needs to process these vectors in order, building up context as it goes.

You’ve used this feature hundreds of times: typing on your phone, and it predicts what you’ll say next.

Type “I’m going to the” and your phone suggests: gym, store, beach…

But how does it know? The key insight: context matters. The prediction for “I’m going to the ___” depends entirely on what came before. If you had typed “I’m running to the ___”, the suggestions would shift toward finish line, bathroom, car.

This is the fundamental challenge with sequential data — each piece of information depends on what came before it.

Quick Check

Which of these tasks benefits MOST from remembering previous information?

Your turn

Choose an answer to check your understanding.

Attempts: 0

The Big Idea: A Network with Memory

Imagine reading a mystery novel. You don’t forget chapter 1 when you reach chapter 5 — you carry information forward. Clues from early chapters help you understand later events.

Recurrent Neural Networks (RNNs) work the same way. They’re neural networks with a “memory” that gets updated as they process each piece of a sequence.

Here’s the simple loop:

🧠 RNN Processing Sequence

Input Sequence

Ilovemy ___

↓

Hidden State (Memory)

↓

Next Word Predictions

30%

love

25%

think

20%

have

15%

Step 1 / 3

What to look for: Step through the words and watch two things: (1) the memory bubble accumulates words as the RNN reads, and (2) the prediction bars shift at each step. After “I,” the model spreads probability across many verbs. After “I love,” it focuses on pronouns and possessives. After “I love my,” pets and family dominate. Each word narrows the possibilities.

Why Context Changes Everything

Traditional neural networks treat each input independently — they have no memory. That’s like trying to understand a sentence by looking at each word in isolation. RNNs solve this by passing information forward through the sequence.

💡 Why Context Matters

Input: "bank"

🚫 Without Context

Financial institution? River edge?

→

✓ With Context

'river bank' vs 'bank account' — context tells you which!

Input: "spring"

🚫 Without Context

A season? A coil? A water source?

→

✓ With Context

'spring break' vs 'spring coil' — context resolves the ambiguity!

Fill in the Blanks

RNNs maintain a state that captures information from previous .

Fill in every blank, then check your answer.

How It Works: The Core Loop

Let’s break down what happens at each step. Don’t worry about the math yet — just the intuition:

At each timestep:

new_memory = combine(current_input, old_memory)
prediction = transform(new_memory)

That’s it! The network learns how to combine and transform through training.

Here’s that loop in action — watch how each step feeds into the next:

🔁 The RNN Loop

📖 Step 1: Read

Read one piece of input

Reading: "I"

Processing word 1 / 3

Ilovemy

Read

What to look for: The loop cycles through four phases for each word: Read (take in the current word), Update (blend it with existing memory), Predict (output a guess for the next word), Repeat (move to the next word). Watch the word tracker at the bottom: after processing “I,” the model predicts “love.” After processing “love,” it predicts “my.” The same four-step cycle runs for every word in the sequence.

In the Forward Pass, we’ll unpack the real equations, including the weight matrices, the tanh activation, and softmax, and trace through the math step by step.

Key insight: At each step, the hidden state $h_t$ depends on two things:

The current input $x_t$ (the word just read)
The previous hidden state $h_{t-1}$ (everything seen before)

This is the recurrent connection — information flows forward through time.

What we skipped on this page:

The vanishing gradient problem. RNNs struggle with long sequences because gradients get multiplied many times during backpropagation. If those multipliers are less than 1, the gradient “vanishes,” becoming too small to learn from. We explore this in detail with interactive visualizations in Backward Pass & BPTT and Vanishing Gradients & Gated RNNs.
The original RNN. Elman, “Finding Structure in Time” (1990) introduced the “simple recurrent network” (now called the Elman network or vanilla RNN). The key idea: feed the hidden state from the previous timestep back as an additional input. This is the architecture you’ll build in the Forward Pass.
LSTM. Hochreiter & Schmidhuber, “Long Short-Term Memory” (1997) solved the vanishing gradient problem by adding gates that control what information to keep, forget, or output. LSTMs dominated sequence modeling for nearly 20 years before transformers arrived. You’ll see them in Vanishing Gradients & Gated RNNs.
How information travels. In Why Transformers Won, you passed a memory through each word, then read an earlier clue directly. Here you will see how an RNN learns to update that memory.

Quick Summary

Concept	What Happens	Intuition
Sequential data	Each input depends on what came before	”I’m going to the ___” needs prior words to predict
Hidden state	Memory vector updated at each step	Like carrying clues forward through a mystery novel
Core loop	Combine(input, memory) → prediction	Two ingredients: what you see now + what you remember
Context	Same word means different things in different sequences	”bank” = river edge or financial institution

Learning Checklist

I can explain why sequential data is harder than independent data
I understand the “mystery novel” analogy for how RNNs carry memory forward
I know the two inputs to each RNN step (current input + previous hidden state)
I can give an example of why context changes a word’s meaning
I understand why traditional neural networks can’t handle sequences well

Ready to see the real equations and watch the computation unfold? Head to the Forward Pass →