The model can’t read characters the way you do. Before anything else, your text is split into tokens — chunks that are often whole words, sometimes word-pieces, sometimes single characters.

This is done with byte-pair encoding (BPE): start from raw bytes and repeatedly merge the most common adjacent pair into a new token. Frequent sequences like " the" become a single token; rare ones get fragmented.

Try it → type in the panel. A few things worth discovering:

• “strawberry” splits into pieces (straw + berry). The model never sees the individual r’s — which is exactly why “how many R’s in strawberry?” is weirdly hard for it. Counting letters isn’t a native operation.
• A leading space is part of the token: " Paris" and "Paris" are different tokens.
• Numbers, code, emoji, and other languages tokenize very differently. Toggle GPT-4 (~100k vocab) vs GPT-4o (~200k) and watch the same text use fewer tokens with the bigger vocabulary — that’s cheaper and better at other languages.

Each token is really just an integer — its row number in the model’s dictionary. That list of integers is the only thing the network receives.

A bare token ID like 2746 means nothing on its own. So the model looks it up in a giant table and swaps it for a long list of numbers — a vector. For GPT-2 small that’s 768 numbers per token; modern models use thousands.

These vectors are learned (in the gradient-descent sense — the How Models Learn track shows the machinery). Tokens used in similar contexts end up near each other in this space, so distance and direction encode meaning. In the panel, each dot is a word; click one to light up its nearest neighbors — dog sits by puppy and kitten, far from algorithm.

Because meaning lives in directions, you can sometimes do arithmetic on words. Toggle the analogy: king − man + woman lands right on queen.

Honest asterisk: this famous example is hand-picked. It only works cleanly when you exclude the input words, and it describes static word vectors — a real LLM’s internal representations are contextual and messier. It’s a beautiful hint that meaning is geometric, not a hard law.

One thing embeddings don’t carry yet is order — “dog bites man” and “man bites dog” would look the same. Models fix that by adding positional information to each vector before the next step.

To understand a word you need context. In “The animal didn’t cross the street because it was too tired,” what does “it” refer to? Self-attention lets each token look at the other tokens and decide how much to pull from each.

Mechanically, every token produces three vectors: a Query (“what am I looking for?”), a Key (“what do I offer?”), and a Value (“what I’ll hand over”). A token’s query is compared against every key; the scores go through a softmax into weights that sum to 1; the output is that weighted blend of Values.

Do it for every pair and you get the attention matrix in the panel — row i is how much token i attends to each other token. Notice the greyed upper triangle: a token may only look backward, never at words it hasn’t generated yet. That causal mask is what makes the model a left-to-right predictor. Hover any cell to read the weight.

Two things to keep in mind as we simplify:

• Real models run many attention heads in parallel (GPT-2 small: 12), each learning a different kind of relationship, plus a feed-forward layer where a lot of the model’s stored knowledge lives.
• This whole block is stacked many times (12 in GPT-2 small; 80+ in frontier models). The token’s vector gets refined at every layer.

After the last block, the model is ready to turn that vector back into a word.

You just saw self-attention — where tokens share context. After attention, each transformer block has a second stage: the feed-forward network (FFN), the dense thinking layer where roughly two-thirds of a model’s parameters live.

That FFN is the entire swap. Flip the toggle in the panel:

• Dense: one big FFN. Every token flows through all of it — all those parameters, every time.
• MoE: the same slot now holds eight separate FFNs — the “experts” — and a router: a single tiny learned layer that scores all eight and sends the token to the top 2. The other six sit idle, costing nothing for this token.

Notice what did not change: the attention layer above is untouched, and so are the embeddings, the norms, and everything downstream. That’s why MoE isn’t a new kind of model — it’s a dense transformer where one sublayer learned to delegate.

One honest nuance: models differ on which blocks get the swap. Mixtral does it in every block; Llama 4 Maverick alternates dense and MoE blocks; DeepSeek-V3 keeps its first three blocks dense. The mechanism is identical everywhere it appears.

So how does that router actually decide? Let’s route some real tokens.

The router is almost embarrassingly simple: one small linear layer. It looks at a token’s vector and produces a score (a logit) for each expert. Then — exactly like the sampling step you know — a softmax turns scores into weights. Mixtral’s recipe: keep the top 2, cut everyone else to −∞ (their weight becomes zero), and blend the two chosen experts’ outputs by their weights:

y = g₁ · Expertᵃ(x) + g₂ · Expertᵇ(x)

Step through the panel, or hit Route them all. Every token gets its own scoring, its own top-2, its own blend — and this happens independently in every MoE block, so the same token consults different experts at different depths.

Then comes the reveal, and it’s the part most people get wrong. When Mixtral’s authors analyzed real routing, they found no topic specialists — no math expert, no biology expert. ArXiv papers and philosophy essays route almost identically. What clusters instead is syntax: punctuation flocks to one expert, identifiers to another, whitespace and keywords elsewhere. Related research found “punctuation experts” and “number experts” — the division of labor is real, just far weirder and lower-level than the name “expert” suggests.

So each token only pays for 2 of 8 experts. Time to see what that buys at frontier scale.

Now the numbers. An MoE model has two sizes, and the gap between them is the whole point:

• Total parameters — every expert in every layer. This is what must sit in memory, and where the knowledge lives.
• Active parameters — the slice one token actually runs through: the shared layers plus its chosen experts. This is what you pay in compute, per token.

Flip through the models in the panel. Mixtral kicked off the open-source MoE era: 46.7B total, 12.9B active — it matched or beat Llama 2 70B with roughly 6× faster inference. DeepSeek-V3 pushed it further: 671B total, 37B active — about 5.5% of the model working per token, with 256 tiny experts per layer instead of 8 big ones. OpenAI’s open-weights gpt-oss-120b activates just 4.4%.

That’s the industry trend in one line: more, smaller experts; lower active fraction — ever more capacity for the same per-token compute. It’s a big part of why training DeepSeek-V3 cost only about $5.6M in GPU time.

(And yes, GPT-4 is in the picker — widely believed to be an MoE, never confirmed. It wears its rumor badge.)

One catch, though. All of this depends on the router spreading work evenly — and left alone, routers don’t.

There’s a trap hiding in the router, and it was spotted in the very first sparse-MoE paper: a rich-get-richer loop. Whichever expert happens to be picked slightly more often gets more training signal, so it improves, so the router picks it even more — until one expert is doing nearly all the work and the rest are dead weight. All that capacity you paid for, wasted.

Try it in the panel: set the mode to “No balancing” and watch the collapse. Then try the two eras of fixes:

• Auxiliary loss (Switch Transformer, 2021): add a small penalty (α ≈ 0.01) whenever token traffic is uneven. Simple, effective — but it tugs against the model’s main objective, which can cost a little quality.
• Bias nudges (DeepSeek-V3, 2024, “auxiliary-loss-free”): no penalty at all. Each expert gets a tiny bias used only when the router ranks experts — nudged down each step it’s overloaded, up when it’s starved. Traffic levels out without ever touching the loss.

This unglamorous plumbing is a real reason MoE took a decade to go mainstream — and a quiet ingredient in why DeepSeek’s models train so cheaply.

With the experts balanced, the block’s job is done. What happens next? Exactly what you already know.

After the last block, the model takes the final token’s vector and scores every token in its vocabulary — that’s ~50,000 raw numbers called logits. A softmax squashes them into probabilities that sum to 100%. Now we have a genuine distribution: “87% chance the next token is Paris, 3% the, …”

How do we actually pick one? That’s what the controls in the panel do:

• Greedy — always take the top token. Safe, but repetitive.
• Temperature — rescales the logits. Drag it down and one bar dominates (conservative, near-deterministic at 0). Drag it up and the bars flatten (more diverse, riskier). Temperature 0 ≠ “more true” — just more predictable.
• Top-p (nucleus) — keep only the smallest set of tokens whose probability adds up to p, then sample from those. The struck-through bars are the tail it’s ignoring.

Press Sample! a few times to feel the randomness — then set temperature to 0 and watch it become deterministic.

Then the crucial move: whatever token we pick gets appended to the sequence, and the whole pipeline runs again to produce the next one. This autoregressive loop — one token at a time — is how a whole paragraph gets written.

And it’s exactly this loop that a reasoning model turns into something more.

“Mixture of Experts” is a great name and a misleading one. It conjures a panel of specialists — a doctor, a lawyer, a mathematician — voting on your question. Almost none of that picture survives contact with the evidence. Tap each card in the panel and see for yourself:

1. “Experts are topic specialists.” Mixtral’s own routing analysis found no subject-matter specialization at all — the real clusters are syntactic (punctuation, numbers, indentation), just as you saw in the routing scene.
2. “MoE is an ensemble of separate models.” The experts are FFN fragments inside every layer; one token weaves a different path through them at every depth. There are no eight little models — which is why 8×7B ≠ 56B.
3. “Only the active experts need to be in memory.” Every expert must be loaded, because any token might need any of them at any layer. Memory scales with total; compute scales with active. MoE models are cheap to run, not small to hold.

And that’s the whole track. The same pipeline you’ve known since the first scroll — tokenize, embed, attend, sample — with one sublayer that learned to delegate. From Mixtral’s 8 experts to DeepSeek’s 256, that single idea is how today’s frontier models got enormous without sending every token the full bill.