How Models Learn.

Strip away the mystique and a neuron is one small formula:

out = activation(w₁·x₁ + w₂·x₂ + b)

It takes its inputs, multiplies each by a weight (how strongly an input contributes), adds a bias (an offset), and passes the total through an activation function (a transformation of that total).

The panel shows one neuron’s opinion about every possible input at once — amber on one side, blue on the other. Drag the knobs:

• The weights tilt the dividing line. Big w₁ means “input 1 decides.”
• The bias slides the line without turning it.
• The activation sets the edge: step is a hard verdict, sigmoid and tanh are graded confidence, ReLU only speaks up on one side.

Two things to hold onto. First, those knobs — the weights and bias — are the neuron’s parameters. When we say a model learns, we mean exactly this: its parameters change. This toy keeps the architecture fixed.

Second, notice what one neuron can’t do: its boundary is always a straight line. However you set the knobs, it can only split the world in two with one cut. Keep that limitation in mind for the next panel.

This three-parameter example is a building block. Larger networks combine such operations with attention, normalization, and other machinery. Parameter counts for undisclosed models should not be treated as known facts.

The panel’s data is the classic XOR problem: blue lives in two opposite quadrants, amber in the other two. No single straight line can separate them — which means the neuron from last panel, alone, cannot solve this. This exact limitation stalled neural-net research for years.

The fix is embarrassingly simple: use more than one, in layers.

Your browser just trained a network with four hidden neurons feeding one output neuron. Look at the little thumbnails — each hidden neuron is still just a straight-line splitter, exactly like the last panel. Click one to see its view of the world, full size.

Then look at the combined map. The output neuron takes those four straight opinions, weighs them (with its own learned weights!), and blends them into a bent boundary that carves out all four quadrants correctly.

That’s the whole trick, and it’s the deepest fact in this track:

• One neuron: a line.
• One layer of neurons + a blend: corners and curves.
• With suitable activations and enough capacity, networks can approximate broad classes of functions. Universal approximation is an existence result under assumptions, not a promise that training finds a good solution.

Depth composes. Early neurons learn simple cuts; later neurons combine cuts into shapes; still-later ones combine shapes into concepts. In the LLM tracks you saw the same story at scale — attention layers stacking 80+ deep, each refining the last layer’s work.

“Learn” implies “improve,” and “improve” implies a score. In machine learning that score is the loss: a single number that says how wrong the model currently is. For the nonnegative loss used here, smaller is better. The entire training process exists to push this one number down.

The panel gives you the simplest possible model — a straight line with two knobs, slope and offset — and eleven data points it should pass through. Every dashed pink bar is one prediction’s miss (the residual). The loss here is mean squared error: square every miss, average them.

Drag the knobs and watch the meter. Two things worth feeling out:

• Squaring punishes big misses hard. One awful prediction hurts more than five slightly-off ones. Get the line badly wrong for the rightmost point and watch the loss spike.
• There’s a floor you can’t reach. The points are noisy, so even the best line can’t hit zero. The meter turns teal when you’re near the best achievable — good is defined relative to the data, not as perfection.

One number to summarize millions of predictions may sound crude, and it is — choosing what to measure is one of the most consequential design decisions in all of AI. A common pretraining loss measures next-token prediction error over a chosen training corpus. Other training stages use other objectives; good loss values do not by themselves establish truthfulness, fairness, or useful behavior.

Now: you can feel that some knob-directions lower the loss and some raise it. You just discovered the gradient — next panel makes it precise.

A separate one-input neuron, fit to twelve data points, has exactly two knobs: a weight w and a bias b. That means we can do something the textbooks usually fake: plot a grid of true loss values across a range of both knobs. That’s the landscape in the panel — dark valley, amber ridges. It is not an artist’s impression; it’s computed live from the actual loss.

Click the plot or use “drop at a corner” to place the ball. At every point, the gradient says which way is uphill — so the ball steps the opposite way. That’s the entire algorithm:

knob ← knob − learning_rate × gradient

Repeat until the valley. That’s the “learning” every tab on this site keeps mentioning. It has a name you’ve now earned: gradient descent.

The learning rate slider sets the step size, and it’s the most temperamental dial in all of deep learning:

• Too small: the ball creeps; training takes forever.
• Just right: a brisk, confident roll to the bottom.
• Too big (try > 3): each step overshoots the valley and the ball rattles itself right out of the bowl. That failure is real, not staged — it’s the same divergence that makes real training runs explode into NaNs.

Honest asterisk: large models can have billions of parameters, and their loss surfaces are not tidy bowls. This two-parameter example computes real gradient steps; it does not capture every difficulty of large-scale training.

Gradient descent needs the gradient — a blame number for every knob. With two knobs you could find it by wiggling each one and re-measuring. With a billion, wiggle-and-check would take geologic time. Backpropagation computes all the blames in one backward sweep, and it’s why deep learning is possible at all.

The panel lays one neuron out as a computational graph — every multiply, add, and squash as its own node. Step through it:

• Forward (blue numbers): the input flows left to right, exactly the arithmetic from Step 1, ending in a loss.
• Backward (pink ∇ numbers): the loss asks the node before it, “how much did you move me?” — and that node asks its inputs, and so on, right to left. Each answer is the node’s local effect multiplied by the blame arriving from ahead of it. Calculus calls this the chain rule; you can read it as blame flows backward, scaled at every step.

Notice the division of labor when you reach the knobs:

• The add node passes blame through untouched.
• The multiply node scales blame by the other input — so a knob that was amplifying a big signal gets more blame than one riding a small signal.

When the sweep finishes, press apply the nudge: every knob moves against its gradient, and the loss drops. Forward, backward, nudge — you just did one lap of the training loop, by hand.

A real training step is this exact walk over a graph with billions of nodes, run by autograd machinery (PyTorch, JAX) instead of your thumb — millions of laps, batched thousands of examples at a time. Same walk. Bigger graph.

Time to let go of the crank. The panel is the whole loop — guess, measure, blame, nudge — running locally in your browser on a network of 105 parameters. Speed depends on your device and settings. The painted map is what the network currently believes; the dots are the truth it’s trying to match.

Press train and watch the belief map organize itself out of nothing:

• XOR falls in a blink — you watched a 4-neuron net do it in Step 2.
• Circle takes a beat longer: the net has to bend a line into a ring.
• Spiral is the honest one. Watch the early epochs flail — the loss curve stalling, the boundary sloshing — before it suddenly catches, and the arms sharpen. That stretch of apparent futility is what most of training feels like at every scale.

Things worth trying:

• Reset and retrain. Same starting weights, same data, same result — training is deterministic here. (Real runs randomize; we seed everything so you and every other reader watch the identical learning curve.)
• Switch datasets to start another controlled run and compare the pattern.

There is no model file behind this panel, no video, no server — just the arithmetic from the last five panels, repeated fast. The small example makes optimization visible. It does not reproduce the data, architecture, objectives, or complete training process of a large assistant.

The loss curve’s shape — fast drop, long grind, occasional sudden improvement — is the same silhouette you saw in the reasoning track’s RL training scene. Different reward, same rolling-downhill machinery.

There’s a cheat that beats every training loss: memorize the answers. A big enough network can wrap a perfect little boundary around every training point — noise, mistakes and all — and drive its training loss to nearly zero while understanding nothing.

The defense is beautifully simple, and it’s the ritual behind every model you use: hide some data. In the panel, the network trains only on the solid dots; the hollow dots are the held-out test set it never sees. Two scores, two very different meanings:

• Train loss — how well it fits what it studied.
• Test loss — how well it generalizes to what it didn’t.

Now break it on purpose. Push the noise slider up and give the huge network its head. Watch the boundary grow ragged islands around individual mislabeled dots — and watch the two losses come apart: train keeps falling, test doesn’t follow. A growing gap can be a sign of overfitting. Sampling and distribution differences also affect the comparison. The model isn’t learning the circle; it’s learning the noise.

Then try the same noise with the tiny network. Its limited capacity can make fitting noise harder; in this example it may score better on held-out points. That is a tendency to inspect, not a universal rule that smaller models generalize better.

The tension is real at every scale: frontier LLMs have enough capacity to memorize their training data verbatim, and a whole discipline — regularization, held-out evals, deduplication — exists to keep them generalizing instead. When you hear “the model was evaluated on held-out benchmarks,” this panel is what that sentence means.