End-to-End Walkthrough

You’ve learned the architecture, training, and inference separately. This page puts it all together: one complete forward pass through a toy transformer, with numbers at every stage.

Interactive: Follow the Data

Below is a working decoder-only transformer (d_model=8, 2 heads, 1 layer, vocab=32). Type a prompt and expand each stage to see the actual values flowing through the model.

Its weights come from a fixed random seed. The calculations let you inspect a forward pass; training is what would make its predictions useful. In the next lab, you’ll run a separate PyTorch model and update its weights on text.

Try it yourself:

1. Expand each stage (click the headers) to see intermediate values
2. Adjust temperature: low (0.1) = confident, high (2.0) = creative
3. Sample tokens to extend the text one word at a time
4. Change the prompt to see how different inputs produce different predictions

Stage by Stage: What’s Happening

Each stage below maps to what you learned in earlier pages. The numbers in the interactive tool above are real, so you can follow along.

Tokenization splits text into integer IDs. “The cat sat on” becomes [0, 1, 2, 3]. The model never sees text, only numbers. (Forgot how? Review tokenization.)

Embedding looks up each ID in a learned table, producing one 8-dim vector per token. At this point, “cat” always maps to the same vector regardless of context. (Forgot how? Review embeddings.)

Positional encoding adds a unique position signal to each embedding so the model knows token order. Without this, “cat sat” and “sat cat” would be identical. (Forgot how? Review positional encoding.)

Self-attention is where tokens communicate. Each token produces Q, K, V vectors, computes attention scores via dot products, applies a causal mask (the lower triangle), normalizes with softmax, and takes a weighted sum of V vectors. The last row of the attention matrix drives the prediction, because it represents the final token looking back at everything before it. (Forgot how? Review self-attention.)

Residual + LayerNorm + FFN stabilizes and transforms. The attention output is added back to the input (residual connection), normalized, then processed by the FFN. The FFN expands to 2x width (8 dims to 16 in our toy model), applies ReLU (zeroing out roughly half the neurons), then contracts back to 8 dims. Why the detour through a wider layer? The expansion gives ReLU room to create sparse activation patterns: different inputs zero out different neurons, producing unique “fingerprints” that encode complex features. The contraction then reads those fingerprints back into the original dimension. (Full explanation with worked example. Forgot residuals? Review layer norm & residuals.)

Prediction takes the last token’s final hidden vector, projects it to vocabulary size (32 logits), divides by temperature, and applies softmax to get probabilities. The highest-probability token is the model’s best guess for what comes next.

Real vs. Toy Model

Our visualization uses a tiny model for clarity. Here’s how it compares to real transformers:

The models share a core data flow: embeddings, attention, per-token transformations, and vocabulary scores. Real models also differ in positional encoding, normalization, activation functions, and how attention heads share information. The toy example makes that common structure small enough to inspect.

Quick Summary

Next: Train Your Own GPT

You’ve traced the forward pass. Now prepare a small text dataset, update the weights, and measure what the model learns in Train Your Own GPT →.