Neural Network Basics

Every symbol refers to a number in the example:

• $x$: the fixed input, 2
• $y$: the target answer, 2
• $w$: the adjustable weight, initially 0.5
• $\hat{y}$ (“y hat”): the model’s prediction
• $L$: the squared-error loss
• $\eta$ (“eta”): the learning rate, 0.1
• $\frac{\partial L}{\partial w}$: the gradient, meaning the sensitivity of the loss to a small change in the weight

The forward pass and loss are:

The loss depends on the prediction, and the prediction depends on the weight. The chain rule multiplies those two sensitivities:

Subtract the gradient times the learning rate. The subscript “new” means after this update:

These sensitivities describe the loss near the current weight. They do not promise that an arbitrarily large step will help: with learning rate 0.3, the weight becomes 1.7, the prediction 3.4, and the loss 1.96.

This optional code needs Python and PyTorch. It implements the same single weight update as the visualization, without a training framework around it:

import torch

x = torch.tensor(2.0)                       # shape: () = one scalar input
target = torch.tensor(2.0)                  # shape: () = one scalar target
w = torch.tensor(0.5, requires_grad=True)   # shape: () = one trainable weight
learning_rate = 0.1

prediction = w * x                         # shape: ()
loss = (prediction - target).square()       # shape: ()
loss.backward()                            # computes gradients, no update yet

print(f"Before: prediction={prediction.item():.2f}, loss={loss.item():.2f}")
print(f"Gradient: {w.grad.item():.2f}")

with torch.no_grad():                      # don't record the update as model math
    w -= learning_rate * w.grad

w.grad = None                             # clear before the next backward pass

with torch.no_grad():                      # evaluation needs no new gradient graph
    next_prediction = w * x
    next_loss = (next_prediction - target).square()

print(f"New weight: {w.item():.2f}")
print(f"After: prediction={next_prediction.item():.2f}, loss={next_loss.item():.2f}")

Expected output:

Before: prediction=1.00, loss=1.00
Gradient: -4.00
New weight: 0.90
After: prediction=1.80, loss=0.04

requires_grad=True marks a tensor for gradient tracking. .item() reads a one-element tensor as a Python number for printing. Gradients accumulate across backward calls, so we clear .grad before the next training step. We also recompute the prediction and loss for each new step.

With many parameters, a PyTorch optimizer such as torch.optim.SGD handles the update. Its zero_grad() clears old gradients, loss.backward() computes new ones, and step() applies them. An optimizer such as Adam uses additional running statistics, but the predict → loss → backward → update loop stays the same.

Here a vector has 2 input features. We process 3 examples together and produce 4 output features per example:

import torch

inputs = torch.tensor([
    [2.0, -1.0],
    [0.0,  3.0],
    [1.0,  1.0],
])                                         # shape: (3, 2) = (batch, input features)
weights = torch.zeros(4, 2)                 # shape: (4, 2) = (output features, input features)
bias = torch.zeros(4)                      # shape: (4,) = (output features,)
scores = inputs @ weights.T + bias         # shape: (3, 4) = (batch, output features)
outputs = torch.relu(scores)               # shape: (3, 4)
print(tuple(outputs.shape))                # (3, 4)

@ means matrix multiplication. .T transposes this two-dimensional weight tensor, turning shape (4, 2) into (2, 4). The shared dimension 2 matches the input feature count. Adding the bias reuses its four numbers for each of the three examples. The activation changes values while preserving shape.

The zero weights make this a shape demonstration with zero outputs. For training a multilayer network, we would initialize weights differently, typically with small random values so neurons can learn different features.

• More weights. With several weights, the gradient has one entry per weight. Each entry asks how the loss changes when that weight moves a little, holding the others fixed. Autograd traces all the paths that use each weight.
• Different losses. Squared error makes our numerical example easy to inspect. For next-token prediction, we’ll use cross-entropy, which penalizes assigning low probability to the correct token. Updating shared weights can help one example and hurt another, so training uses many examples and checks performance on held-out data.
• More than one update. Repeat the forward pass after every update to get a fresh prediction, loss, and gradient. An old gradient describes the old weight, not every future position.