Transformer
You now have every building block:
| Piece | What it does | Where you learned it |
|---|---|---|
| Embeddings | Turn token IDs into dense vectors | Foundations |
| Self-Attention | Let every token look at every other token | Attention |
| Multi-Head Attention | Run multiple attention perspectives in parallel | Attention |
| Positional Encoding | Stamp position information onto each vector | Attention |
| Layer Norm & Residuals | Keep activations stable across deep stacks | Foundations |
| Cross-Attention | Let one sequence attend to another | Attention |
One piece is missing: the Feed-Forward Network, the “thinking” step that follows attention’s “gathering” step. Once you have that, every component snaps together into a full transformer.
What You’ll Learn in This Chapter
You’ll assemble the final transformer pieces: the Feed-Forward Network, the repeatable transformer block, and the main encoder-decoder, encoder-only, and decoder-only variants. You’ll also see how transformers train, generate text, use the KV cache, and carry real numbers from raw text to a next-token prediction. The chapter ends with a Python lab: train a small GPT from random weights, evaluate it, and save and reload your model.
Chapter Roadmap
| Module | What You’ll Do |
|---|---|
| Architecture Overview | Meet the FFN, assemble a transformer block, compare three architecture variants |
| Training | Understand next-token prediction, cross-entropy loss, teacher forcing, and the training loop |
| Inference | See how autoregressive generation works, why the KV cache matters, and how sampling strategies shape output |
| End-to-End Walkthrough | Trace real numbers through every stage of a decoder-only transformer |
| Train Your Own GPT | Prepare text, train and evaluate a small GPT, save a checkpoint, and generate text |
Start with the Architecture Overview →