The model can’t read characters the way you do. Before anything else, your text is split into tokens — chunks that are often whole words, sometimes word-pieces, sometimes single characters.

This is done with byte-pair encoding (BPE): start from raw bytes and repeatedly merge the most common adjacent pair into a new token. Frequent sequences like " the" become a single token; rare ones get fragmented.

Try it → type in the panel. A few things worth discovering:

• “strawberry” splits into pieces (straw + berry). The model never sees the individual r’s — which is exactly why “how many R’s in strawberry?” is weirdly hard for it. Counting letters isn’t a native operation.
• A leading space is part of the token: " Paris" and "Paris" are different tokens.
• Numbers, code, emoji, and other languages tokenize very differently. Toggle GPT-4 (~100k vocab) vs GPT-4o (~200k) and watch the same text use fewer tokens with the bigger vocabulary — that’s cheaper and better at other languages.

Each token is really just an integer — its row number in the model’s dictionary. That list of integers is the only thing the network receives.

A bare token ID like 2746 means nothing on its own. So the model looks it up in a giant table and swaps it for a long list of numbers — a vector. For GPT-2 small that’s 768 numbers per token; modern models use thousands.

These vectors are learned (in the gradient-descent sense — the How Models Learn track shows the machinery). Tokens used in similar contexts end up near each other in this space, so distance and direction encode meaning. In the panel, each dot is a word; click one to light up its nearest neighbors — dog sits by puppy and kitten, far from algorithm.

Because meaning lives in directions, you can sometimes do arithmetic on words. Toggle the analogy: king − man + woman lands right on queen.

Honest asterisk: this famous example is hand-picked. It only works cleanly when you exclude the input words, and it describes static word vectors — a real LLM’s internal representations are contextual and messier. It’s a beautiful hint that meaning is geometric, not a hard law.

One thing embeddings don’t carry yet is order — “dog bites man” and “man bites dog” would look the same. Models fix that by adding positional information to each vector before the next step.

To understand a word you need context. In “The animal didn’t cross the street because it was too tired,” what does “it” refer to? Self-attention lets each token look at the other tokens and decide how much to pull from each.

Mechanically, every token produces three vectors: a Query (“what am I looking for?”), a Key (“what do I offer?”), and a Value (“what I’ll hand over”). A token’s query is compared against every key; the scores go through a softmax into weights that sum to 1; the output is that weighted blend of Values.

Do it for every pair and you get the attention matrix in the panel — row i is how much token i attends to each other token. Notice the greyed upper triangle: a token may only look backward, never at words it hasn’t generated yet. That causal mask is what makes the model a left-to-right predictor. Hover any cell to read the weight.

Two things to keep in mind as we simplify:

• Real models run many attention heads in parallel (GPT-2 small: 12), each learning a different kind of relationship, plus a feed-forward layer where a lot of the model’s stored knowledge lives.
• This whole block is stacked many times (12 in GPT-2 small; 80+ in frontier models). The token’s vector gets refined at every layer.

After the last block, the model is ready to turn that vector back into a word.

After the last block, the model takes the final token’s vector and scores every token in its vocabulary — that’s ~50,000 raw numbers called logits. A softmax squashes them into probabilities that sum to 100%. Now we have a genuine distribution: “87% chance the next token is Paris, 3% the, …”

How do we actually pick one? That’s what the controls in the panel do:

• Greedy — always take the top token. Safe, but repetitive.
• Temperature — rescales the logits. Drag it down and one bar dominates (conservative, near-deterministic at 0). Drag it up and the bars flatten (more diverse, riskier). Temperature 0 ≠ “more true” — just more predictable.
• Top-p (nucleus) — keep only the smallest set of tokens whose probability adds up to p, then sample from those. The struck-through bars are the tail it’s ignoring.

Press Sample! a few times to feel the randomness — then set temperature to 0 and watch it become deterministic.

Then the crucial move: whatever token we pick gets appended to the sequence, and the whole pipeline runs again to produce the next one. This autoregressive loop — one token at a time — is how a whole paragraph gets written.

And it’s exactly this loop that a reasoning model turns into something more.

Everything up to here — tokenize, embed, attention, sample — describes a pretrained model. And a pretrained model has a surprising flaw: it doesn’t answer questions. It only continues text. Ask it “How do I make pizza?” and it might reply with more questions, because that’s a plausible continuation.

Flip through the stages in the panel to see the same model become an assistant:

• Base (pretrained): trained only to predict the next token on the whole internet. It autocompletes — it doesn’t follow instructions.
• + Instruction tuning (SFT): fine-tune (the same gradient-descent loop from How Models Learn, on new data) on many curated instruction → good response examples. Now it learns the assistant format and actually does what’s asked.
• + RLHF: humans rank competing responses; a reward model learns their preferences; the model is then optimized (PPO, or newer methods like DPO) to produce responses people prefer. This is what makes it feel helpful, harmless, and honest.

Try the “you be the labeler” step — picking the better of two answers is exactly the human-preference signal RLHF is built from.

That’s the whole Chat LLM: the shared spine you already know, plus these two post-training stages. A reasoning model goes one step further — it adds the long thinking loop from the Reasoning track.