BERT — Bidirectional Encoder Representations from Transformers

Two-panel comic: 'GPT's Exam' shows a robot student at a desk with a cardboard shield blocking the right side of the sentence, only seeing 'The ___', sweating and guessing 'Dog? Fish? Car?', captioned 'Can only read left to right'; 'BERT's Exam' shows the same robot with no shield, reading the full sentence 'The ___ sat on the mat because it was purring', confidently writing 'cat!', captioned 'Reads the whole sentence'

What You’ll Learn

You’ll learn why bidirectional attention makes BERT strong at understanding tasks, and how Masked Language Modeling trains it to use both left and right context. You’ll also see how the [CLS] token, fine-tuning, and BERT’s contrast with GPT fit together.

See the Difference: BERT vs GPT

Click any word to mask it. Compare how a model that sees the whole sentence (BERT) vs. one that sees only what came before (GPT) predicts the masked word:

Masked Language Modeling

Click any token to mask it

Click any word above to mask it and compare BERT vs GPT predictions

What to look for:

Mask a word in the middle of the sentence — BERT uses both sides, GPT can only use the left
Mask the last word — both models perform similarly (no right context to use)
Notice how much BERT’s predictions improve when right context is informative

Why Looking Both Ways Matters

Try masking “mat” in the sentence above. With only left context (“The cat sat on the”), you might guess “mat”, “floor”, or “chair.” But with right context too (“because it was dirty”), “mat” becomes much more likely — you need both sides!

BERT’s innovation: bidirectional attention. Every token sees every other token — no causal mask. This makes it powerful for understanding tasks, where you have the full input available.

Architecture

BERT is simply the encoder stack from the original Transformer — no decoder, no causal masking:

Stage	What Happens
Input	`[CLS]` + tokens + `[SEP]`
Embedding	Token + Position + Segment embeddings
Encoder	12 (or 24) bidirectional self-attention layers
Output	Contextualized representation per token

Model	Layers	Hidden	Heads	Parameters
BERT-base	12	768	12	110M
BERT-large	24	1024	16	340M

Pre-training: Masked Language Modeling (MLM)

BERT’s training objective: randomly mask 15% of tokens, then predict the originals. Let’s trace through a concrete example.

Input: "The [MASK] sat on the mat" (we masked “cat”)

BERT embeds each token, adds positional encodings, and runs 12 bidirectional attention layers. Because attention is bidirectional, the [MASK] position attends to all other tokens — “The”, “sat”, “on”, “the”, “mat.” By the final layer, the representation at position 2 encodes rich context from both sides. This is projected to vocabulary logits (30,522 scores), and softmax assigns probabilities: maybe “cat” gets 0.67, “dog” gets 0.12, “bird” gets 0.08. The loss is −log(0.67) ≈ 0.4, and gradients flow back to improve the representations.

The 80/10/10 Strategy

Of the 15% selected tokens:

Treatment	Percentage	Example
Replace with `[MASK]`	80%	“The [MASK] sat on the mat”
Replace with random word	10%	“The banana sat on the mat”
Leave unchanged	10%	“The cat sat on the mat”

Quick Check

Why does BERT replace some masked tokens with random words instead of always using [MASK]?

Your turn

Choose an answer to check your understanding.

Attempts: 0

Fill in the Blanks

BERT uses attention (not causal), meaning every token can see every token.

Fill in every blank, then check your answer.

Next Sentence Prediction (NSP)

MLM teaches BERT about token-level context, but many downstream tasks involve comparing sentences — “Is this passage relevant to this question?” or “Does sentence A contradict sentence B?” BERT’s second pre-training objective tackles this: given two sentences, predict whether the second actually follows the first in the original text.

Input	Label
`[CLS]` “The cat sat on the mat” `[SEP]` “It was purring softly”	IsNext
`[CLS]` “The cat sat on the mat” `[SEP]` “Stock prices rose sharply”	NotNext

The [CLS] token’s representation is used for this binary classification. This teaches BERT about sentence-level relationships — useful for tasks like question answering and natural language inference.

Note: Later work (RoBERTa) showed that NSP may not actually help much. RoBERTa removes it entirely and trains better. But understanding NSP helps explain BERT’s original design choices, especially the [CLS] and [SEP] tokens.

The [CLS] Token

BERT prepends a special [CLS] token to every input. Its final representation acts as a sentence-level summary:

Because [CLS] attends to all other tokens via self-attention, it aggregates information from the entire sentence — perfect for classification tasks.

Fine-tuning BERT

BERT follows a two-stage paradigm:

Stage 1: Pre-train on massive unlabeled text (MLM objective)
Stage 2: Fine-tune on a small labeled dataset for your specific task

Task	Input	Output From	Example
Sentiment	Single sentence	[CLS]	“Great movie!” → Positive
NER	Single sentence	Each token	”Alice went to Paris” → PER, O, O, LOC
Q&A	Question + Passage	Each token	Find start/end of answer span
Similarity	Two sentences	[CLS]	“I like cats” ≈ “Cats are great”

BERT vs GPT

Aspect	BERT	GPT
Architecture	Encoder-only	Decoder-only
Attention	Bidirectional (sees all)	Causal (sees only left)
Pre-training	Masked LM (fill in blanks)	Next token prediction
Strength	Understanding tasks	Generation tasks
Can generate?	Not directly	Yes, autoregressively

The Practical Impact

BERT dominated NLP benchmarks from 2018-2020. But the tide shifted:

GPT-3 (2020) showed that large decoder-only models can do understanding tasks too — via in-context learning
Modern LLMs (GPT-4, Claude, LLaMA) are all decoder-only but handle both understanding and generation

BERT-style models remain useful for:

Efficient classification and search (smaller, faster)
Sentence embeddings (e.g., all-MiniLM, E5)
Token-level tasks like NER

RoBERTa (2019)

Removed NSP (next sentence prediction) objective
Trained longer with more data
Dynamic masking (re-mask each epoch)
Consistently outperforms BERT

DistilBERT (2019)

40% smaller, 60% faster
Retains 97% of BERT’s performance
Uses knowledge distillation

DeBERTa (2020)

Disentangled attention (separate content and position)
Enhanced mask decoder
State-of-the-art on many benchmarks

Modern Embedding Models

E5 and GTE: BERT-style models optimized for embeddings
all-MiniLM: Compact model for semantic search
These power most RAG (retrieval-augmented generation) systems today

Quick Summary

Concept	What Happens	Intuition
Bidirectional attention	Every token attends to every other token — no causal mask	Like reading a whole sentence before filling in a blank, not guessing word by word
Masked Language Modeling	15% of tokens are masked; BERT predicts the originals	A fill-in-the-blank exam that forces deep contextual understanding
80/10/10 strategy	Masked tokens are replaced with [MASK] (80%), random words (10%), or kept (10%)	Prevents BERT from only learning to “spot the [MASK] token”
[CLS] token	Special token whose final representation summarizes the whole input	An attentive reader who listens to every word and writes a summary
Fine-tuning	Pre-trained BERT is adapted to specific tasks with a small labeled dataset	A well-read student who quickly learns a new subject
Encoder-only	No decoder, no autoregressive generation	Built for understanding, not for writing

Key References

BERT. Devlin et al., “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2018) introduced masked language modeling and the pre-train/fine-tune paradigm that dominated NLP for two years. The key insight: bidirectional context from MLM produces better representations than left-to-right language modeling.
RoBERTa. Liu et al., “RoBERTa: A Robustly Optimized BERT Pretraining Approach” (2019) showed BERT was significantly undertrained. Removing NSP, training longer with more data, and using dynamic masking improved performance across all benchmarks.
DistilBERT. Sanh et al., “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter” (2019) demonstrated that knowledge distillation can compress BERT to 40% fewer parameters while retaining 97% of its performance, making it practical for production deployment.
DeBERTa. He et al., “DeBERTa: Decoding-enhanced BERT with Disentangled Attention” (2020) separated content and position into two vectors with their own attention matrices, improving performance on tasks where position matters (e.g., NER, span extraction).

Learning Checklist

I can explain why bidirectional attention helps understanding tasks but prevents autoregressive generation
I understand BERT’s Masked Language Modeling objective and the 80/10/10 masking strategy
I can describe how the [CLS] token aggregates sentence-level information through self-attention
I understand the pre-train then fine-tune paradigm and how BERT adapts to different downstream tasks
I can compare BERT and GPT architecturally and explain when each approach is more appropriate
I know why modern LLMs shifted toward decoder-only architectures despite BERT’s success

Next: GPT

While BERT excels at understanding, GPT excels at generation. Next: GPT — Decoder-Only Transformers →