Foundations

Two problems sit between raw text and a first language model: turning words into numbers, then processing sequences of those numbers. This chapter tackles both.

The first modules cover the input pipeline and the smallest neural network building block:

• Tokenization — splitting text into pieces from a fixed vocabulary
• Embeddings — converting those pieces into number vectors
• Neural Network Basics: transform vectors with weights, bias, and activations, then follow one training update and try PyTorch tensors

The next five modules build and break the first sequence model, motivating everything that comes later:

• RNN Introduction — the idea of processing sequences with memory
• RNN Forward Pass — computing outputs step by step, with real numbers
• RNN Backward Pass & BPTT — how the network learns from mistakes
• Vanishing Gradients & Gated RNNs — why RNNs fail on long sequences, and partial fixes (LSTM, GRU)
• Layer Norm & Residuals — stability tricks that make deep networks trainable

By the end, you’ll understand exactly what problem attention needs to solve.

Read the RNN animations and explanations in order. The full BPTT derivation and LSTM/GRU implementation details are available for a closer look; you can return to them after your first training run. The later small GPT lab uses PyTorch’s automatic differentiation to calculate gradients.