GPT — Generative Pre-trained Transformer

Two-panel comic: 'Raw GPT' shows a robot buried in books labeled 'All of the Internet', asked 'What is the capital of France?' but spewing a ticker tape of text '...is a question frequently asked in geography textbooks published by...', captioned 'Predicts the next word. That's all.'; 'After Human Feedback' shows the same robot now wearing a headset and 'Helpful' badge, calmly answering 'Paris.' while a human holds up RLHF good/bad cards, captioned 'Now it actually helps.'

What You’ll Learn

You’ll learn how GPT uses next-token prediction to generate text, and why scale makes that simple objective surprisingly powerful. You’ll also see how sampling controls, scaling laws, in-context learning, and RLHF shape raw prediction into useful assistant behavior.

The Big Idea: Just Predict the Next Word

GPT’s training objective is the simplest possible:

Given all previous tokens, predict the next one.

That’s it. No masking, no fill-in-the-blank, no special objectives. Just relentlessly predict the next token, billions of times, on massive text corpora.

The joint probability of a sequence factorizes as a product of conditional probabilities — each token is predicted given all previous tokens:

$P(x_1, x_2, \ldots, x_n) = \prod_{i=1}^{n} P(x_i \mid x_1, \ldots, x_{i-1})$

This is the chain rule of probability applied to language. GPT learns to model each conditional $P(x_i \mid x_1, \ldots, x_{i-1})$ using a decoder-only transformer.

The remarkable discovery: this simple objective, at sufficient scale, produces general intelligence.

Architecture

GPT is the decoder stack from the Transformer — with causal (masked) self-attention:

Stage	What Happens
Input	Token sequence (e.g., “The cat sat on the”)
Embedding	Token + Position embeddings
Decoder layers	N layers of causal self-attention + FFN
Output projection	Hidden state → vocabulary probabilities
Prediction	Sample next token from distribution

Model	Year	Layers	d_model	Params	Context
GPT-1	2018	12	768	117M	512
GPT-2	2019	48	1600	1.5B	1024
GPT-3	2020	96	12288	175B	2048
GPT-4	2023	?	?	~1T?	128K

Interactive: Generation in Action

Experiment with autoregressive generation. Adjust temperature and top-k to see how sampling strategies change the output:

Autoregressive Generation

Prompt:

Temperature: 1.0

Top-k: 10

The cat ▌

Next token distribution:

"was"

30.3%

"sat"

21.1%

"ran"

13.5%

"is"

9.8%

"and"

5.2%

"small"

4.8%

"on"

4.3%

"your"

4.0%

"hat"

3.5%

"dog"

3.4%

"the"

0.0%

"a"

0.0%

Try: Set temperature to 0.1 (deterministic) vs 2.0 (random). Set top-k to 1 (greedy) vs 30 (diverse).

What to look for: Generate a few tokens at low temperature (0.1), then reset and try high temperature (2.0). Low temperature produces predictable, repetitive text. High temperature produces surprising, sometimes nonsensical text. Then experiment with top-k: setting it to 1 forces greedy decoding regardless of temperature.

Understanding the Controls

Temperature scales the logits before softmax — low temperature makes the model more deterministic, high temperature makes it more creative:

Temperature	Distribution	Effect
T = 0.1	Very peaked	Almost always picks the top token (deterministic)
T = 1.0	Standard	Samples naturally from learned distribution
T = 2.0	Flattened	More diverse, surprising choices

Top-k limits sampling to the top k most likely tokens, filtering out low-probability noise.

Top-p (nucleus sampling) includes the smallest set of tokens whose cumulative probability exceeds $p$ . Unlike top-k which always considers the same number of tokens, top-p is adaptive — it considers fewer tokens when the model is confident and more when it’s uncertain.

Context	Top-k (k=5)	Top-p (p=0.9)
“The capital of France is”	Always 5 tokens	Maybe 1-2 tokens (high confidence)
“I feel really”	Always 5 tokens	Maybe 10+ tokens (uncertain)

This is why most production LLMs (including ChatGPT and Claude) use top-p rather than top-k.

Quick Check

If you set temperature to 0.1 and top-k to 1, what type of generation do you get?

Your turn

Choose an answer to check your understanding.

Attempts: 0

Scaling Laws: More Compute = Better Model

GPT-3’s paper revealed that model performance follows predictable power laws. In plain terms: if you want a better model, just make it bigger, feed it more data, and give it more compute. The improvement is smooth and predictable — no guesswork needed.

Factor	Practical Meaning
More parameters	10x more params gives roughly 16% loss reduction
More data	10x more data gives roughly 20% loss reduction
More compute	10x more compute gives roughly 11% loss reduction

The loss $L$ follows power-law relationships with each factor:

Factor	Scaling Relationship
Parameters ( $N$ )	$L(N) \propto N^{-0.076}$
Data ( $D$ )	$L(D) \propto D^{-0.095}$
Compute ( $C$ )	$L(C) \propto C^{-0.050}$

These are empirical power laws — the exponents were measured by training many models at different scales and fitting the curves. The key insight is that loss decreases as a smooth power law, which means you can predict how much improvement to expect before spending the compute.

Emergent Abilities

At certain scales, new capabilities appear suddenly — they’re nearly absent in smaller models:

Ability	Appears Around	Example
Few-shot learning	~1B params	Learning from examples in the prompt
Arithmetic	~10B params	”What is 47 × 83?”
Code generation	~50B params	Writing working Python functions
Chain-of-thought	~100B params	”Let me think step by step…”
Self-correction	~100B+ params	”Wait, that’s wrong. Let me reconsider…”

In-Context Learning: GPT’s Superpower

GPT-3’s most surprising ability: learning from examples in the prompt without updating weights.

Prompting Style	Example
Zero-shot	”Translate to French: Hello world” → “Bonjour le monde”
One-shot	”Good morning → Bonjour” then “Hello world →” → “Bonjour le monde”
Few-shot	Multiple examples then query → model identifies the pattern

The model doesn’t update its parameters — it uses the attention mechanism to identify the pattern in the prompt and apply it. This is a form of meta-learning that emerges at scale.

Quick Check

During in-context learning (few-shot prompting), what happens to the model's weights?

Your turn

Choose an answer to check your understanding.

Attempts: 0

Fill in the Blanks

GPT uses self-attention, meaning each token can only attend to tokens. This enables autoregressive generation one token at a time.

Fill in every blank, then check your answer.

From GPT to ChatGPT: RLHF

Pre-trained GPT is a great text predictor but not a great assistant. Reinforcement Learning from Human Feedback (RLHF) bridges this gap:

Step	Process	Result
1	Pre-train on text (next-token prediction)	Raw language model
2	Supervised fine-tuning (SFT) on conversations	Follows instructions
3	Train reward model from human preferences	Knows what “good” looks like
4	Optimize with PPO/DPO using reward model	Helpful, harmless, honest

This process transforms a raw text predictor into ChatGPT, Claude, or other chat models.

The Open-Source Revolution (2023+)

Model	Organization	Params	Notable Feature
LLaMA	Meta	7-70B	RoPE, RMSNorm, SwiGLU
LLaMA 2	Meta	7-70B	GQA, longer context
Mistral 7B	Mistral AI	7B	Sliding window attention
Mixtral 8x7B	Mistral AI	47B (12B active)	Mixture of Experts
Gemma	Google	2-7B	RMSNorm, GeGLU
Qwen 2.5	Alibaba	0.5-72B	Multilingual, long context

These models approach or match GPT-3.5 quality while being freely available. The combination of architectural innovations (RoPE, GQA, SwiGLU, MoE) with efficient training recipes has democratized large language models.

Quick Summary

Concept	Details
Architecture	Decoder-only transformer with causal masking
Training	Next-token prediction on massive text
Key innovation	Scale + simple objective = general intelligence
Sampling	Temperature, top-k, top-p control generation diversity
Emergent abilities	Few-shot learning, reasoning, code generation
RLHF	Aligns raw LLM with human preferences

Key References

GPT-1. Radford et al., “Improving Language Understanding by Generative Pre-Training” (2018) showed that pre-training a decoder-only transformer on next-token prediction, then fine-tuning on downstream tasks, outperformed task-specific architectures. The “pre-train then fine-tune” paradigm started here.
GPT-3 and in-context learning. Brown et al., “Language Models are Few-Shot Learners” (2020) scaled to 175B parameters and discovered that large enough models can learn new tasks from just a few examples in the prompt, with no weight updates. This “in-context learning” ability was the key surprise.
Scaling laws. Kaplan et al., “Scaling Laws for Neural Language Models” (2020) showed that loss follows smooth power laws in parameters, data, and compute. This made model performance predictable and justified the massive compute investments that followed.
InstructGPT / RLHF. Ouyang et al., “Training language models to follow instructions with human feedback” (2022) described the three-stage pipeline (pre-train → SFT → RLHF) that turned raw GPT-3 into a helpful assistant. This is the recipe behind ChatGPT.
DPO. Rafailov et al., “Direct Preference Optimization: Your Language Model is Secretly a Reward Model” (2023) simplified RLHF by eliminating the separate reward model, optimizing human preferences directly. Now widely used as a simpler alternative to PPO.

Learning Checklist

I can explain GPT’s next-token prediction objective and why it produces general capabilities at scale
I understand how temperature, top-k, and top-p control generation diversity and can describe the tradeoffs
I can describe scaling laws and why bigger models with more data reliably improve
I understand in-context learning (zero-shot, one-shot, few-shot) and that no weights are updated during it
I can explain the RLHF pipeline that transforms a raw language model into a helpful assistant
I can compare GPT and BERT architecturally and explain why modern LLMs are decoder-only

Next: An Optional Training Lab, Then Agents

Use the training loop you learned to adapt an existing model in Fine-tune an Open Model (Optional) →. The lab starts with pretrained weights and saves a small LoRA adapter.

You can also continue directly to LLM Agents →, where a language model becomes part of a loop that calls tools and responds to their results.