Embeddings

What You’ll Learn

You’ll learn why one-hot vectors are large and semantically blind, and how embeddings replace them with compact learned vectors. You’ll also see how lookup tables turn token IDs into vectors a transformer can use.

The simplest way to represent a token as numbers is a one-hot vector: a sparse vector of all zeros with a single 1. This works, but it has serious problems. Embeddings are the elegant fix.

The Problem with One-Hot

The most naive approach: give each token a unique position in a vector of zeros, then put a 1 at that position:

One-Hot Encoding

Step 1: Pick a token

Choose one token. A one-hot vector gives every vocabulary token its own slot.

This has three problems:

1. Dimension Explosion

With a real vocabulary of 50,000 tokens, each one-hot vector has 50,000 dimensions. That’s 200KB per token. A 1,000-token sequence would need 200MB just for inputs.

Representation	Dimensions	Memory per Token
One-hot (50K vocab)	50,000	200 KB
Embedding (256-dim)	256	1 KB
Savings	195× smaller	200× less memory

2. No Semantic Information

One-hot treats every pair of tokens as equally different. The distance between any two one-hot vectors is always the same:

Pair	Semantic Relationship	Distance
cat ↔ dog	Both mammals, pets	same
cat ↔ fish	Less related	same
cat ↔ quantum	Completely unrelated	same

Every token is equally “far” from every other token. There’s no concept of similarity.

3. Wasteful Computation

When you multiply a weight matrix by a one-hot vector, most of the computation is wasted multiplying by zeros. The result is just one column of the matrix. Why do 50,000 multiplications when you could just look up the right column?

The Solution: Embedding Lookup

An embedding matrix stores a dense vector for each token. Instead of multiplying by a one-hot vector, we simply look up the row. Think of it like a dictionary where each token ID maps to a small, meaningful vector.

Try it yourself. Click a token below to see how its ID selects one row from the embedding matrix:

Token ID

“the” → Row 0

Embedding matrix (click any row)

Copied vector (8 dims)

0.05 d0

-0.12 d1

0.41 d2

0.08 d3

-0.35 d4

0.19 d5

0.02 d6

-0.08 d7

What to look for: Each token ID points to exactly one row. The output embedding is just that row copied out.

Quick Check

What does an embedding lookup do?

Your turn

Choose an answer to check your understanding.

Attempts: 0

How Embeddings Learn

The embedding matrix is not hand-designed. It’s initialized randomly and learned during training, just like any other weight matrix.

During backpropagation, gradients flow back through the network into the embedding vectors. But there’s a key difference: only the tokens that appeared in the current batch get updated.

What happens	Embedding rows affected
Batch contains tokens [5, 10, 15]	Only rows 5, 10, 15 get gradient updates
Token 5 appears 3 times in batch	Row 5 accumulates 3 gradient contributions
Token 42 not in batch	Row 42 stays unchanged this step

After seeing millions of sentences like “the cat sat”, “the dog sat”, “the cat ran”:

Word Pair	Similarity	Why
cat ↔ dog	High	Both appear in “the ___ sat”
sat ↔ ran	High	Both appear in “the cat ___“
cat ↔ ran	Lower	Different grammatical roles

Tokens that appear in similar contexts tend to get similar embeddings. This is the distributional hypothesis: “You shall know a word by the company it keeps.”

Fill in the Blanks

Embeddings place tokens with similar close together in vector space, because the model learns that they can often be for each other.

Fill in every blank, then check your answer.

Quick Check

Why use learned embeddings instead of one-hot vectors?

Your turn

Choose an answer to check your understanding.

Attempts: 0

Embeddings in Transformers

In a Transformer, embeddings are the first layer. Step through the pipeline to see how raw text becomes the vectors the model works with:

Embedding Pipeline

Tokenize

Input

The

cat

sat

What to look for: Step through all four stages. Raw words become token IDs, each ID gets looked up in the embedding matrix, and a position vector is added so the model knows order. These combined vectors enter the transformer layers.

Weight Tying

A clever trick used in GPT, BERT, and many others: the same embedding matrix is used for both input and output. At the input, you look up a row to get a vector. At the output, you compare the hidden state with token embeddings to get a score for every token in the vocabulary.

This halves the embedding parameters and ensures input/output live in the same semantic space. A token’s embedding is also its “target vector” for prediction.

Quick Summary

Concept	What Happens	Intuition
One-hot encoding	Each token becomes a sparse vector with a single 1	Like giving every student a unique ID number, with no info about who they are
Embedding lookup	Token ID indexes directly into a dense matrix row	Like looking up a student’s profile: compact and informative
Learning embeddings	Gradients update only the rows used in each batch	Tokens in similar contexts get nudged toward similar vectors
Weight tying	Same matrix for input lookup and output prediction	The model asks “does my prediction look like this word’s meaning?”

What we skipped on this page:

word2vec in depth. Mikolov et al., “Efficient Estimation of Word Representations in Vector Space” (2013) showed that simple neural networks trained on next-word prediction learn vectors with algebraic structure (king - man + woman = queen). The follow-up paper “Distributed Representations of Words and Phrases and their Compositionality” introduced negative sampling, which made training practical.
GloVe. Pennington et al., “GloVe: Global Vectors for Word Representation” (2014) achieved similar results by factorizing the word co-occurrence matrix directly, showing that the same geometric structure emerges from a purely statistical approach.
Static vs. contextual embeddings. word2vec and GloVe produce one vector per word, so “bank” has the same embedding whether it means a riverbank or a financial institution. Contextual embeddings (ELMo, 2018; then BERT and GPT) produce different vectors for the same word depending on surrounding context. The embedding layer you learned about here is the starting point; the transformer layers above it produce the contextual representations.
Embedding dimensionality. Why 768? Larger embeddings capture more information but need more data to train and more memory to store. GPT-2 uses 768, BERT-base uses 768, GPT-3 uses 12,288. The right size depends on model capacity and training data.

Learning Checklist

I can explain the three problems with one-hot vectors (dimension explosion, no semantics, wasteful computation)
I understand that embedding lookup uses a token ID to select one learned row
I can describe how embeddings learn semantic relationships through training (distributional hypothesis)
I can explain weight tying and why sharing the embedding matrix between input and output makes sense

Next Steps

Now we know how tokens become vectors. Next we need the smallest neural network move: turning one vector into another with learned weights. Neural Network Basics →