Optional: Fine-Tune an Open Model

Complete Train Your Own Small GPT first. This lab assumes you can explain targets, gradients, validation loss, and checkpoints. You can skip this optional module and continue to Agents.

Give an Existing Model a Narrow Task

Suppose you want short course glossary answers. A pretrained model may answer correctly but write three paragraphs when you need one sentence. We can give it examples of the desired response and use the same forward → loss → backward → update loop from the training chapter.

The previous lab started from random weights. This one starts from SmolLM2-135M-Instruct, a public checkpoint with a tokenizer and chat template provided by its authors. Its model card documents English as its primary language, so the bundled examples use English. See the model card and usage example.

LoRA adds small trainable matrices to selected layers while holding the original weights fixed. It is like attaching editable notes to an existing textbook: the original text stays in place, and the notes change how you use it. Unlike written notes, these additions are numerical transformations learned through gradients. They still need the original model to run.

The bundled examples are deliberately small. They check that preparation, training, evaluation, and saving work. They do not demonstrate better generalization or a reliable glossary assistant.

1. Check the Local Pipeline

Use the same downloadable lab and open its implementation directory. Install the optional dependencies:

uv sync --extra finetune
uv run --extra finetune finetune.py --smoke-test --out-dir runs/finetune-smoke

The smoke test creates a tiny random model and tokenizer locally, trains for three updates, saves an adapter, and reloads it. It downloads no model weights. Dependency installation may need an internet connection. Random-model output is not a language-quality result.

Open runs/finetune-smoke/metrics.json. reload_loss_matches should be true: loading the saved adapter should reproduce the final validation loss within numerical tolerance. The script also records whether the fixed greedy samples match after reloading.

2. Read the Examples Before Training

The training file is data/finetune-train.jsonl; the held-out file is data/finetune-val.jsonl. JSONL stores one JSON object per line:

{"prompt":"Explain attention in one sentence.","completion":"Attention lets a token combine information from other allowed tokens using learned weights."}

Treat the completion as the answer you want the model to learn. The script formats both fields with the model’s own chat template. It includes the prompt as context but scores only the assistant answer and its ending tokens; prompt and padding positions get label -100, which the loss ignores. Transformers’ causal language model shifts predictions and labels internally, so the script does not shift them a second time.

For example, in a five-token answer, five answer positions contribute to the loss. Padding added to match another example’s length contributes nothing. This prevents a run from appearing better merely by predicting padding or copying the prompt.

The files contain separate prompts, and the script rejects overlapping prompts across the two splits. That check cannot detect every paraphrase or duplicated source. When you replace the teaching data, separate related examples by source or topic before writing the files. Keep a final test set untouched if you repeatedly tune choices against validation.

prefix = tokenizer.apply_chat_template(
    [{"role": "user", "content": prompt}],
    tokenize=True,
    add_generation_prompt=True,
)
tokens = tokenizer.apply_chat_template(
    [
        {"role": "user", "content": prompt},
        {"role": "assistant", "content": completion},
    ],
    tokenize=True,
    add_generation_prompt=False,
)
labels = [-100] * len(prefix) + tokens[len(prefix):]

3. Train the Adapter

Run the public model after the smoke test succeeds:

uv run --extra finetune finetune.py --steps 60 --device cpu --out-dir runs/finetune

The first run downloads the base checkpoint and tokenizer. The script loads full-precision weights; it does not use quantization or an external training service. Use --device mps or --device cuda only when that device is available in your PyTorch installation. If your machine cannot hold the model, the offline smoke test still lets you inspect the workflow.

The script prints the number of trainable parameters. It adds LoRA matrices to the query and value projections you learned in Self-Attention, then updates only those additions:

config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    target_modules=["q_proj", "v_proj"],
    r=8,
    lora_alpha=16,
    lora_dropout=0.0,
    bias="none",
)
model = get_peft_model(base_model, config)

r=8 sets the width of the small added matrices. Keep these settings fixed for your first experiment. You can inspect the ordinary PyTorch training loop in the script without learning a separate training framework. The PEFT quicktour documents this adapter setup and save/load API.

To use your own examples:

uv run --extra finetune finetune.py \
  --train-file data/my-train.jsonl \
  --val-file data/my-val.jsonl \
  --steps 60 \
  --out-dir runs/my-task

Choose a new output directory for each experiment so the script preserves earlier adapters. Keep the base model, held-out file, and generation settings fixed when comparing runs.

4. Compare the Same Examples Before and After

The script evaluates every example in the validation file before any update and again after training. Neither evaluation updates weights. It averages loss over scored answer tokens, so padded positions do not distort the comparison.

Inspect three saved files:

A lower held-out loss means the model assigned more probability to these reference answers. It does not guarantee factual correctness or the requested format. Read each saved output and record whether it is correct, concise, and relevant. An answer can improve in one respect and worsen in another.

Your comparison task: Make a table with one row per saved prompt and columns for before, after, correctness, and one-sentence format. Include failures. With your own larger dataset, inspect additional held-out prompts rather than choosing only the best examples.

5. Reload the Result

The training script saves the adapter, tokenizer, and base-model reference under runs/finetune/adapter. It then reloads them and checks validation loss again. Generate from that saved result in a new process:

uv run --extra finetune finetune.py \
  --adapter runs/finetune/adapter \
  --prompt 'Explain attention in one sentence.' \
  --max-new-tokens 40

The adapter is the learned addition, so loading it also requires its original base model. The metadata records the resolved model revision and hashes of your data files. Preserve those references with your experiment report. The offline smoke adapter also depends on the smoke-base directory created beside it; keep that directory when moving the smoke-test output.

This optional script saves an adapter for evaluation and generation. The core small GPT lab demonstrates continuing an interrupted training run with optimizer state.

Quick Summary

Fine-tuning changes the model’s responses through training. Next, see how an application can use those responses to choose tools and act in Agents →.