You’ll learn why character-level and word-level tokenization each break down, and how subword tokenization finds a useful middle ground. You’ll also see how tokenizers turn text into integer IDs and why that choice affects what the model can learn.
Before a model can process text, it needs to break it into pieces, called tokens. Each token gets mapped to an integer ID. An ID is a vocabulary address, not a measure of meaning.
Clear the 3 tokenizer puzzles below (· marks a leading space):
Level 1 · The Hidden Space TrickAdd a single space at the very start of the text and watch the first token’s ID change!
Loading the GPT-2 tokenizer…
Quick presets:
The Tokenization Trade-off
There are three strategies, each with a fundamental trade-off:
What if we could get the best of both? Subword tokenization splits text into pieces that are often larger than characters but smaller than words. The tokenizer learns useful pieces from training data.
“unhappiness” → ["un", "happiness"] — the model learns that “un” means negation!
Input
Tokens
Why it works
”unhappiness”
["un", "happiness"]
Learns morphemes
”playing”
["play", "ing"]
Shares verb stems
”cryptocurrency”
["crypt", "o", "currency"]
Handles novel words
Subword tokenization gives us a moderate vocabulary, usually tens of thousands of tokens, with reasonable sequence lengths. Many tokenizers can represent any text by falling back to bytes or characters; others still keep an [UNK] token as a fallback. This family of methods powers GPT, BERT, LLaMA, T5, and most modern language models.
The key insight: common words often stay whole (“the”, “hello”, “cat” may be single tokens), while rare or novel words get split into familiar pieces.
🤔Quick Check
Why is subword tokenization preferred over word-level or character-level?
Your turn
Choose an answer to check your understanding.
Try another option
Think about the trade-offs in the table above...
Correct!
Subword tokenization hits the sweet spot: moderate vocabulary size, reasonable sequence lengths, and useful pieces that can be shared across related words. Character-level is often too long; word-level struggles with rare words.
Answer: B
Subword tokenization hits the sweet spot: moderate vocabulary size, reasonable sequence lengths, and useful pieces that can be shared across related words. Character-level is often too long; word-level struggles with rare words.
Attempts: 0
The Complete Pipeline
Regardless of which tokenization strategy you use, the pipeline is always the same:
"I love cats" → tokenize → ["I", "love", "cats"] → lookup → [42, 891, 3017]
Split the text into tokens (subwords, in practice)
Map each token to an integer ID using a fixed vocabulary table
Feed those IDs into the model (next stop: Embeddings)
That’s it. For understanding how transformers work, this is all you need to know about tokenization: text goes in, a sequence of integer IDs comes out, and each ID will get looked up in an embedding table.
✍️Fill in the Blanks
Subword tokenization splits rare words into pieces while keeping common words as tokens.
Fill in every blank, then check your answer.
A blank is still empty.
Fill in every blank before checking your answer.
Not quite.
Try another answer, or reveal the solution.
Correct.
One accepted answer is shown in each blank.
Answer revealed.
The blanks now show the answers.
Quick Summary
Concept
What Happens
Intuition
Character-level
Split into individual characters
Tiny vocab, but sequences are very long and meaning is lost
Word-level
Split on whitespace/punctuation
Natural units, but rare words need [UNK]
Subword
Split into learned pieces between characters and words
Common words stay whole; rare words split into known pieces
Token IDs
Each token maps to an integer
The model only sees numbers, not text
Going Deeper
The sections below are optional. Everything above is sufficient for understanding the rest of the course. But if you’re curious about how subword tokenizers actually learn their vocabularies, read on.
Byte Pair Encoding (BPE)
BPE and BPE-like tokenizers are widely used in GPT-style models and many others. The idea is simple:
Training: four steps on repeat:
Start with small units, such as characters or bytes, as your vocabulary
Count every adjacent pair in the training corpus
Merge the most frequent pair into a new token
Repeat until you reach the desired vocabulary size
Watch it happen. Click Play or Step → to walk through BPE training on the corpus “low” (x5), “lower” (x2), “newest” (x6), “widest” (x3):
BPE Training: Learning Merge Rules
1
2
3
4
5
Merge 1 / 5
Corpus (token sequences x frequency)
x5
low
x2
lower
x6
newest
x3
widest
Step 1: Count adjacent pairs
Click Step → or Play to count pairs
Vocabulary (10 tokens)
deilnorstw
What to look for: In each round, the pair counts reveal which two tokens appear next to each other most often. That pair gets fused into a single new token, the corpus is rewritten, and the process repeats. After a few merges, “low” is a single token, and common endings like “est” have been discovered automatically. Rare words still decompose into known pieces.
Applying BPE to New Text
To tokenize a new word, apply the learned merges in priority order (the order they were learned during training):
How does BPE handle a word it has never seen during training (like 'cryptocurrency')?
Your turn
Choose an answer to check your understanding.
Try another option
BPE can fall back to smaller units before applying merges...
Correct!
BPE starts from small units such as characters or bytes, then applies learned merge rules. A never-before-seen word like 'cryptocurrency' can still be split into familiar pieces like 'crypt' + 'o' + 'currency'. Byte-level BPE can represent any text because every byte is covered.
Answer: B
BPE starts from small units such as characters or bytes, then applies learned merge rules. A never-before-seen word like 'cryptocurrency' can still be split into familiar pieces like 'crypt' + 'o' + 'currency'. Byte-level BPE can represent any text because every byte is covered.
Attempts: 0
WordPiece
WordPiece is similar to BPE but with a key difference:
Algorithm
Merge criterion
BPE
Most frequent pair
WordPiece
Pair that maximizes likelihood of the data
This means WordPiece tends to prefer pairs that co-occur more than you would expect from the individual pieces alone.
The WordPiece merge criterion:
pair∗=argmaxpairP(a)⋅P(b)P(merged)
P(merged): how often the merged token appears in the corpus
P(a)⋅P(b): how often you’d expect the pair if a and b occurred independently
A high ratio means the pair co-occurs far more than chance, a strong signal that they form a meaningful unit.
The ## Convention
BERT uses ## to mark continuation subwords (not the start of a word):
Word
Tokens
Meaning
”playing”
["play", "##ing"]
”play” starts the word, “ing” continues it
”unhappy”
["un", "##happy"]
”un” starts, “happy” continues
”unbelievable”
["un", "##believ", "##able"]
Three pieces
The ## prefix lets the model distinguish between “play” as a standalone word and “play” as the start of “playing”.
Models reserve special tokens for structural purposes:
Token
Purpose
When it’s used
[PAD]
Padding
Making all sequences in a batch the same length
[UNK]
Unknown
Fallback for out-of-vocabulary tokens (rare with BPE)
[CLS]
Classification
BERT uses this position for sentence-level predictions
[SEP]
Separator
Between sentence pairs (e.g., question + passage)
[MASK]
Masking
BERT’s masked language model training
<|endoftext|>
End of text
GPT’s end-of-document marker
Tokenizer
Used By
Vocab Size
Boundary convention
BPE
GPT-2, RoBERTa
50,257
Ġ (space prefix)
WordPiece
BERT
30,522
## (continuation)
SentencePiece
T5, LLaMA
Variable
▁ (space prefix)
Tiktoken
OpenAI GPT models
Varies by encoding
Byte-level BPE, spaces included in tokens
Space Handling: Two Approaches
Approach
Example
Who uses it
Space prefixĠ / ▁
”Hello world” → ["Hello", "Ġworld"]
GPT-2, T5, LLaMA
Continuation prefix##
”playing” → ["play", "##ing"]
BERT
What we skipped on this page:
The original BPE paper. Sennrich et al., “Neural Machine Translation of Rare Words with Subword Units” (2016) adapted the byte pair encoding compression algorithm for NLP tokenization. This helped popularize subword tokenization in neural language models, including GPT-2.
WordPiece origins. Schuster & Nakajima, “Japanese and Korean Voice Search” (2012) introduced WordPiece for speech recognition vocabulary. It was later adopted for BERT. Roughly speaking, WordPiece chooses pieces that improve data likelihood, while BPE uses frequent pair merges.
Tokenization-free models. Recent research explores byte-level models that avoid learned subword tokenizers (e.g., ByT5, MegaByte). They handle languages and scripts more uniformly, but produce much longer sequences and shift the cost into computation.
Learning Checklist
I can explain the trade-offs between character-level, word-level, and subword tokenization
I understand that tokenization converts text into a sequence of integer IDs
I know that common words often stay as single tokens while rare words get split into subword pieces
I can explain why subword tokenization is common in modern language models
Next Steps
Now we know how text becomes a sequence of token IDs. But a bare integer like 3017 doesn’t tell the model anything about what “cats” means. Next, we’ll see how each token ID gets transformed into a rich vector that captures semantic meaning: Embeddings →.
