Attention

RNNs process words one at a time, so distant words can barely influence each other. Attention fixes this: every word looks directly at every other word, regardless of distance. This chapter builds the mechanism from scratch.

• Self-Attention — the core idea: queries, keys, values, and the dot-product score that decides “who should I pay attention to?”
• Multi-Head Attention — running multiple attention perspectives in parallel so the model can track grammar, meaning, and coreference at the same time
• Positional Encoding — how to give attention a sense of word order, since it treats input as a set by default
• Seq2Seq Attention — where the idea originated: letting a decoder look back at all encoder states during translation

By the end, you’ll have every piece needed to assemble a full transformer.