Positional Encoding

$x_{\text{input}} = \text{token\_embedding} + \text{positional\_encoding}$

The positional encoding vector has the same dimension as the token embedding ($d_{\text{model}}$), so the two can simply be added element-wise.

$PE_{(\text{pos}, 2i)} = \sin\left(\frac{\text{pos}}{10000^{2i/d}}\right)$ $PE_{(\text{pos}, 2i+1)} = \cos\left(\frac{\text{pos}}{10000^{2i/d}}\right)$

Where:

• $\text{pos}$: the position in the sequence (0, 1, 2, …)
• $i$: the dimension index (0, 1, 2, … up to $d/2$)
• $d$: the model dimension ($d_{\text{model}}$)
• Even dimensions use sine, odd dimensions use cosine
• The denominator $10000^{2i/d}$ creates exponentially decreasing frequencies as the dimension $i$ increases

def sinusoidal_positional_encoding(max_len, d_model):
    PE = np.zeros((max_len, d_model))
    position = np.arange(max_len)[:, np.newaxis]

    # Different frequency for each dimension pair
    div_term = np.exp(np.arange(0, d_model, 2) * (-np.log(10000.0) / d_model))

    PE[:, 0::2] = np.sin(position * div_term)  # Even dims: sin
    PE[:, 1::2] = np.cos(position * div_term)  # Odd dims: cos

    return PE

Other Position Encoding Methods

Learned embeddings: BERT and GPT-2 learn a $(max\_len, d_{model})$ parameter matrix, giving each position a trainable vector. Slightly more flexible, but can’t handle sequences longer than max_len.

RoPE: Rotary Position Embedding

Sinusoidal encoding adds position information before attention. What if position information lived inside the attention score instead? RoPE rotates query and key vectors by an angle proportional to their position. The dot product of two rotated vectors naturally depends on their relative distance.

What to look for: in step 3, two pairs of tokens at the same distance (positions 2,5 and positions 10,13) produce the same dot product. RoPE makes attention scores depend on how far apart tokens are, not where they sit in absolute terms.

Used by LLaMA, LLaMA 2, Mistral, Gemma, and most modern open-source LLMs.

Why RoPE won:

1. Encodes relative position directly in the Q·K dot product, not as an additive offset
2. With NTK-aware scaling or YaRN, handles sequences 4-8x longer than training
3. No extra parameters (fixed mathematical functions, like sinusoidal)

Key Equations Reference

Sinusoidal Positional Encoding:

$PE_{(\text{pos}, 2i)} = \sin\left(\frac{\text{pos}}{10000^{2i/d_{\text{model}}}}\right)$ $PE_{(\text{pos}, 2i+1)} = \cos\left(\frac{\text{pos}}{10000^{2i/d_{\text{model}}}}\right)$

Input to Transformer:

$x = \text{Embedding}(\text{token}) + \text{PE}(\text{position})$

RoPE:

$q'_m = q_m \cdot e^{im\theta}, \quad k'_n = k_n \cdot e^{in\theta}$

$q'_m \cdot k'_n = q_m \cdot k_n \cdot e^{i(m-n)\theta}$

Key References

• Sinusoidal positional encoding. Vaswani et al., “Attention Is All You Need” (2017), Section 3.5. The authors chose fixed sinusoidal functions over learned embeddings and found similar performance, but sinusoidal PE can extrapolate to longer sequences.

• RoPE. Su et al., “RoFormer: Enhanced Transformer with Rotary Position Embedding” (2021) introduced rotary position embedding, encoding relative position directly in the Q·K dot product. Now the default in LLaMA, Mistral, Gemma, and most modern open-source LLMs.

• ALiBi. Press et al., “Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation” (2022) showed that simply subtracting a linear distance penalty from attention scores provides strong length extrapolation without any learned or fixed position embeddings. Used in BLOOM.