Seq2Seq Attention

Two-panel comic: 'The Bottleneck' shows a robot cramming a long French text scroll into a tiny 'Context Vector' capsule while words spill everywhere; 'The Fix' shows a robot with a spotlight aimed at 'MONDE' on a rack of French words while translating 'WORLD', with caption 'Look back at everything'

What You’ll Learn

You’ll learn how an encoder-decoder model translates one sequence into another, and why squeezing a whole sentence into one vector creates a bottleneck. You’ll also see how cross-attention lets the decoder look back at every encoder state and reveal word-level alignment patterns.

You already know self-attention: every token attends to every other token in the same sequence. But where did this idea come from? It started with a simpler problem: how do you translate a long sentence without losing information?

How Encoder-Decoder Translation Works

Translation needs to read an entire input sentence, then produce an output sentence in a different language. The earliest neural approach split this into two parts: an encoder that reads the input word by word, and a decoder that generates the output word by word.

Step through the animation to see the full pipeline:

The encoder reads the input sentence one word at a time, left to right. Each word updates the hidden state. By the end, the final hidden state h3 contains a summary of the entire input.

What to look for: the encoder processes words left to right, accumulating information into a hidden state. At the end, that final state becomes the context vector, the only thing passed to the decoder. The decoder then generates output words one at a time, consulting only this context vector. Step 4 reveals the problem: what happens when the sentence gets longer?

The Bottleneck

Toggle between a short and long sentence to see the problem more clearly:

Seq2Seq Bottleneck

With a short sentence, the context vector has plenty of capacity. Each word gets well represented.

What to look for: toggle to the long sentence and watch the per-word capacity bar shrink. The context vector doesn’t grow with the input; the same fixed-size vector must encode everything, so each word gets less space.

The context vector is always the same size. With 3 words, there’s plenty of room. With 8 words, each word gets a fraction of the capacity, and early words fade as the encoder overwrites them with newer information. For long sentences, details are inevitably lost.

Quick Check

What problem does attention solve in seq2seq models?

Your turn

Choose an answer to check your understanding.

Attempts: 0

The Fix: Look Back at Everything

Instead of one context vector for the entire output, compute a different context for each decoder step by looking back at all encoder states. At each step, the model assigns an attention weight to every encoder state (“how relevant is this input word right now?”), then blends all the encoder states together using those weights.

Click through decoder steps below. Watch how the attention weights shift as each French word is generated:

Seq2Seq Cross-Attention Alignment

English → French Translation

Source (EN):

The 60%

cat 13%

sat 8%

on 5%

the 7%

mat 7%

Target (FR):

Decoder step 1/7: generating "Le"

At each decoder step, the model attends back to the source sentence to find the most relevant words. Notice how "Le" aligns to "The", the semantically corresponding word.

What to look for:

When generating a target word, attention focuses on the most relevant source word(s)
The attention pattern isn’t always left-to-right: it can reorder words across languages
Each decoder step creates a different blend of source information

The result is a weighted sum, a custom-built context vector tailored to exactly what the decoder needs at this step. For translating “world” to “monde”, attention focuses mostly on the encoder state where “world” was encoded, not on all states equally.

Worked Example: Computing Attention Weights

Suppose we’re translating “I love cats” → “J’aime les chats” and the decoder has just output “J’aime”. Now it needs to generate the second word.

Recall from the encoder-decoder pipeline: the encoder processed each input word and produced a hidden state at each position (h1, h2, h3). The decoder RNN also has its own hidden state, s, which summarizes what it has generated so far. This decoder state acts as the query: “given what I’ve produced so far, what source information do I need next?”

Step through the four operations that turn these vectors into a prediction:

The decoder state acts as a query: "What source information do I need right now?" The dot product measures compatibility. Higher score means stronger alignment. Here, the decoder aligns most with "love" (0.46) and least with "cats" (−0.23, pointing away).

What to look for: in Step 1, the dot products reveal which encoder states match the decoder’s query. In Step 2, softmax turns those raw scores into weights that sum to 1. In Step 3, the weighted sum blends encoder states into a context vector dominated by the highest-weighted word. Step 4 closes the loop: the context vector feeds into the final prediction.

Fill in the Blanks

In attention, the decoder state acts as a , the encoder states act as keys, and the context vector is a sum of the encoder states.

Fill in every blank, then check your answer.

Bahdanau (Additive) Attention

The original attention mechanism (Bahdanau et al., 2014) scores compatibility by feeding the decoder state and each encoder state through a small neural network. Think of it like a matchmaking service: the decoder describes what it needs, each encoder state describes what it has, and the network scores how well they match.

$e_{t,i} = v^T \tanh(W_s s_{t-1} + W_h h_i)$ $\alpha_{t,i} = \frac{\exp(e_{t,i})}{\sum_j \exp(e_{t,j})}$ $c_t = \sum_i \alpha_{t,i} \cdot h_i$

Luong (Multiplicative) Attention

A simpler variant (Luong et al., 2015) skips the neural network and measures compatibility using dot products. If two vectors point in a similar direction, their dot product is high, meaning they’re compatible. This is faster and, in many cases, works just as well.

Variant	Score function	Notes
Dot	$e_{t,i} = s_t^T h_i$	Simplest, requires same dimensions
General	$e_{t,i} = s_t^T W h_i$	Learnable, allows different dimensions
Concat	$e_{t,i} = v^T \tanh(W[s_t; h_i])$	Same as Bahdanau

The dot product variant became the basis for self-attention in Transformers.

Python Implementation

def bahdanau_attention(s_prev, encoder_outputs, Ws, Wh, v):
    """Bahdanau (additive) attention."""
    src_len = encoder_outputs.shape[0]
    scores = np.zeros(src_len)
    for i in range(src_len):
        combined = np.tanh(Ws @ s_prev + Wh @ encoder_outputs[i])
        scores[i] = v @ combined
    weights = softmax(scores)
    context = weights @ encoder_outputs
    return context, weights

def luong_attention(s_curr, encoder_outputs, W=None):
    """Luong (multiplicative) attention."""
    if W is not None:
        scores = encoder_outputs @ W @ s_curr  # General
    else:
        scores = encoder_outputs @ s_curr       # Dot
    weights = softmax(scores)
    context = weights @ encoder_outputs
    return context, weights

Key Equations Reference

Concept	Equation
Context vector	$c_t = \sum_{i=1}^{T_x} \alpha_{t,i} \cdot h_i$
Attention weights	$\alpha_{t,i} = \text{softmax}(e_{t,i})$
Bahdanau score	$e_{t,i} = v^T \tanh(W_s s_{t-1} + W_h h_i)$
Luong dot score	$e_{t,i} = s_t^T h_i$

Key References

Sequence-to-sequence learning. Sutskever et al., “Sequence to Sequence Learning with Neural Networks” (2014) showed that an LSTM encoder-decoder could translate sentences end-to-end. This is the vanilla seq2seq model whose bottleneck problem motivated attention.
Encoder-decoder with GRU. Cho et al., “Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation” (2014) independently proposed the encoder-decoder framework and introduced the GRU cell.
Additive (Bahdanau) attention. Bahdanau et al., “Neural Machine Translation by Jointly Learning to Align and Translate” (2014) introduced attention for seq2seq models. The alignment visualizations in the paper are some of the most cited figures in deep learning.
Multiplicative (Luong) attention. Luong et al., “Effective Approaches to Attention-based Neural Machine Translation” (2015) simplified the scoring function to dot products and compared multiple variants. The dot-product variant became the basis for transformer attention.

Cross-Attention vs. Self-Attention

Seq2seq attention lets the decoder attend to the encoder: one sequence looking at another. This is called cross-attention. You’ve already seen the other variant, self-attention, where every token attends to tokens in the same sequence.

	Cross-Attention (this page)	Self-Attention
What attends to what	Decoder looks at encoder (two sequences)	Each token looks at all tokens (one sequence)
Question it answers	”What input word is relevant to my output?"	"What other words in my sentence are relevant to me?”
Where it’s used	Translation, summarization	Every transformer layer

The core mechanism is identical: query-key matching followed by a weighted sum of values. The only difference is where Q, K, and V come from. In self-attention, all three come from the same sequence. In cross-attention, Q comes from one sequence and K, V from another.

Quick Check

In a French-to-English translation, the attention weights form a roughly diagonal pattern. What does this tell you?

Your turn

Choose an answer to check your understanding.

Attempts: 0

Quick Summary

Concept	What Happens	Intuition
Vanilla seq2seq	Encoder compresses entire input into one fixed vector; decoder reads only that	Like summarizing a novel in a single tweet: something has to give
Bottleneck problem	Longer inputs lose more information in the fixed-size vector	100 words sharing 512 floats means ~5 floats per word
Attention	Decoder computes a fresh context vector at each step by weighting all encoder states	Like re-reading the relevant parts of the source for each word you translate
Cross vs. self-attention	Cross: decoder attends to encoder. Self: every token attends to its own sequence	Same mechanism, different source of Q, K, V

Learning Checklist

I can explain the bottleneck problem in vanilla seq2seq and why it gets worse with longer inputs
I understand how attention computes a different context vector for each decoder step
I can trace through a worked example: dot products, softmax, weighted sum
I can interpret an attention alignment matrix and explain what diagonal vs. non-diagonal patterns mean
I understand how cross-attention relates to self-attention (same mechanism, different Q/K/V sources)

Next: The Full Transformer

You’ve now seen the full attention toolkit: self-attention within a sequence, multiple heads for parallel perspectives, positional encoding for word order, and cross-attention between sequences. Time to see how all the pieces fit together. Architecture Overview →