Mixture of Experts.

The model can’t read characters the way you do. Before anything else, your text is split into tokens — chunks that are often whole words, sometimes word-pieces, sometimes single characters.

This is done with byte-pair encoding (BPE): start from raw bytes and repeatedly merge the most common adjacent pair into a new token. Frequent sequences like " the" become a single token; rare ones get fragmented.

Try it → type in the panel. A few things worth discovering:

• Try “strawberry”, with and without a leading space. The split depends on the tokenizer. Tokens do not usually correspond one-to-one with letters; that can make character-level tasks awkward, but it is not a complete explanation of every counting error.
• A leading space is part of the token: " Paris" and "Paris" are different tokens.
• Numbers, code, emoji, and other languages tokenize very differently. Toggle GPT-4 (~100k vocab) vs GPT-4o (~200k) and compare the counts. A larger vocabulary need not use fewer tokens for every string, and token count alone does not determine capability or cost.

Each token is really just an integer — its row number in the model’s dictionary. These IDs select token embeddings; the model also uses positional information and, in some systems, other inputs.

A bare token ID like 2746 means nothing on its own. So the model looks it up in a giant table and swaps it for a long list of numbers — a vector. For GPT-2 small that’s 768 numbers per token; modern models use thousands.

These vectors are learned (in the gradient-descent sense — the How Models Learn track shows the machinery). Relationships can appear in learned representations, but distance depends on the model, layer, and comparison. In this illustrative panel, each dot is a word; click one to light up its nearest neighbors — dog sits by puppy and kitten, far from algorithm.

Because meaning lives in directions, you can sometimes do arithmetic on words. Toggle the analogy: king − man + woman points toward queen in the authored example.

Honest asterisk: this famous example is hand-picked. It only works cleanly when you exclude the input words, and it describes static word vectors — a real LLM’s internal representations are contextual and messier. It’s a beautiful hint that meaning is geometric, not a hard law.

One thing embeddings don’t carry yet is order — “dog bites man” and “man bites dog” would look the same. Models incorporate positional information; some add position embeddings, while others encode relative positions within attention.

To understand a word you need context. In “The animal didn’t cross the street because it was too tired,” what does “it” refer to? Self-attention lets each token look at the other tokens and decide how much to pull from each.

Mechanically, every token produces three vectors: a Query (“what am I looking for?”), a Key (“what do I offer?”), and a Value (“what I’ll hand over”). A token’s query is compared against every key; the scores go through a softmax into weights that sum to 1; the output is that weighted blend of Values.

Do it for every pair and you get the attention matrix in the panel — row i is how much token i attends to each other token. Notice the greyed upper triangle: a token may only look backward, never at words it hasn’t generated yet. That causal mask is what makes the model a left-to-right predictor. Hover any cell to read the weight.

Two things to keep in mind as we simplify:

• Real models run many attention heads in parallel (GPT-2 small: 12), each learning a different kind of relationship, plus a feed-forward layer that applies further learned transformations. Knowledge is not neatly isolated in just one component.
• This whole block is stacked many times (12 in GPT-2 small; a model-dependent number in larger systems). The token’s vector gets refined at every layer.

After the last block, the model is ready to turn that vector back into a word.

You just saw self-attention — where tokens share context. After attention, each transformer block has a second stage: the feed-forward network (FFN), the a learned transformation with a parameter share that depends on architecture.

That FFN is the entire swap. Flip the toggle in the panel:

• Dense: one big FFN. Every token flows through all of it — all those parameters, every time.
• MoE: the same slot now holds eight separate FFNs — the “experts” — and a router: a single tiny learned layer that scores all eight and sends the token to the top 2. The other six skip their expert arithmetic for this token; storing weights and moving data still have costs.

Notice what did not change: the attention layer above is untouched, and so are the embeddings, the norms, and everything downstream. That’s why MoE isn’t a new kind of model — it’s a dense transformer where one sublayer learned to delegate.

One honest nuance: models differ on which blocks get the swap. Mixtral does it in every block; Llama 4 Maverick alternates dense and MoE blocks; DeepSeek-V3 keeps its first three blocks dense. The mechanism is identical everywhere it appears.

So how does that router actually decide? Let’s route an illustrative token sequence.

The router is almost embarrassingly simple: one small linear layer. It looks at a token’s vector and produces a score (a logit) for each expert. Then — exactly like the sampling step you know — a softmax turns scores into weights. Mixtral’s recipe: keep the top 2, cut everyone else to −∞ (their weight becomes zero), and blend the two chosen experts’ outputs by their weights:

y = g₁ · Expertᵃ(x) + g₂ · Expertᵇ(x)

Step through the panel, or hit Route them all. Every token gets its own scoring, its own top-2, its own blend — and this happens independently in every MoE block, so the same token consults different experts at different depths.

Then comes the reveal, and it’s the part most people get wrong. When Mixtral’s authors analyzed real routing, they did not find strong domain-based specialization in the comparisons shown. That result is specific to the studied model and analysis. Some different text domains had similar routing distributions. Their examples highlighted syntax: punctuation flocks to one expert, identifiers to another, whitespace and keywords elsewhere. Related research found “punctuation experts” and “number experts” — the division of labor is real, just far weirder and lower-level than the name “expert” suggests.

So each token only pays for 2 of 8 experts. Time to see what that buys at frontier scale.

An MoE model has two useful counts:

• Total parameters: the full collection of learned weights, including experts that may be inactive for a particular token.
• Active parameters: the subset involved in processing that token, under the model’s routing design.

The picker links each published figure to its source. Mixtral 8×7B has about 46.7B total and 12.9B active parameters; DeepSeek-V3 reports 671B and 37B. The percentages help compare sparsity, not prove equal quality or serving cost.

Unused experts skip their feed-forward arithmetic for that token, but their weights must still be available somewhere. Memory traffic, communication, batching, hardware, and routing affect real performance. Offloading can trade memory pressure for latency. “5% active” does not mean “5% of the bill.”

The GPT-4 option is an explicitly labeled historical rumor. Its numbers are unconfirmed and should not be used as a measured comparison.

Also be careful with cost headlines: DeepSeek-V3’s reported training-compute estimate has a defined scope. It is not a complete budget for the research, data, infrastructure, and experiments that produced the system.

Next: how do we keep a router from sending nearly everything to one expert?

There’s a trap hiding in the router, and it was spotted in the very first sparse-MoE paper: a rich-get-richer loop. Whichever expert happens to be picked slightly more often gets more training signal, so it improves, so the router picks it even more — until one expert is doing nearly all the work and the rest are dead weight. All that capacity you paid for, wasted.

Try it in the panel: set the mode to “No balancing” and watch the collapse. Then try the two eras of fixes:

• Auxiliary loss (Switch Transformer, 2021): add a small penalty (α ≈ 0.01) whenever token traffic is uneven. Simple, effective — but it tugs against the model’s main objective, which can cost a little quality.
• Bias nudges (DeepSeek-V3, 2024, “auxiliary-loss-free”): a bias-based balancing mechanism. Each expert gets a tiny bias used only when the router ranks experts — nudged down each step it’s overloaded, up when it’s starved. Traffic levels out without making that main balancing mechanism a loss penalty. The report still includes a small complementary sequence-wise auxiliary loss.

This plumbing affects both utilization and learning. The animated traffic is a teaching model, not measured DeepSeek training behavior.

With the experts balanced, the block’s job is done. What happens next? Exactly what you already know.

After the last block, the model takes the final token’s vector and scores every token in its vocabulary — about 50,000 scores in GPT-2, and a different count in other models. Those scores are called logits. A softmax squashes them into probabilities that sum to 100%. The panel computes a distribution from authored example scores, not a live model prediction: “87% chance the next token is Paris, 3% the, …”

How do we actually pick one? That’s what the controls in the panel do:

• Greedy — always take the top token. Deterministic for fixed scores, but not guaranteed correct or safe.
• Temperature — rescales the logits. Drag it down and one bar dominates (conservative, near-deterministic at 0). Drag it up and the bars flatten (more diverse, riskier). Temperature 0 does not mean “more true” — just more predictable.
• Top-p (nucleus) — keep only the smallest set of tokens whose probability adds up to p, then sample from those. The struck-through bars are the tail it’s ignoring.

Press Sample! a few times to feel the randomness — then set temperature to 0 and watch it become deterministic.

Then the crucial move: whatever token we pick gets appended to the sequence, and the whole pipeline runs again to produce the next one. This autoregressive loop — one token at a time — is how a whole paragraph gets written.

And it’s exactly this loop that a reasoning model turns into something more.

“Mixture of Experts” is a great name and a misleading one. It conjures a panel of specialists — a doctor, a lawyer, a mathematician — voting on your question. Almost none of that picture survives contact with the evidence. Tap each card in the panel and see for yourself:

1. “Experts are topic specialists.” Mixtral’s routing analysis did not find strong domain-based specialization in the comparisons shown — the real clusters are syntactic (punctuation, numbers, indentation), just as you saw in the routing scene.
2. “MoE is an ensemble of separate models.” The experts are FFN fragments inside every layer; one token weaves a different path through them at every depth. There are no eight little models — which is why 8×7B ≠ 56B.
3. “Only the active experts need to be in memory.” Experts must be available when needed; systems can distribute or offload weights, with memory, communication, and latency trade-offs. Memory scales with total; compute scales with active. Sparse routing reduces selected arithmetic; overall cost still depends on the serving system.

And that’s the whole track. The same pipeline you’ve known since the first scroll — tokenize, embed, attend, sample — with one sublayer that learned to delegate. From Mixtral’s 8 experts to DeepSeek’s 256, that single idea is how today’s frontier models got enormous without sending every token the full bill.