Inside the Black Box.

Imagine five useful signals and only two dimensions to carry them. Should a network drop some signals, or represent them in overlapping directions? Toy models of superposition make that question small enough to investigate.

This panel trains a 15-parameter implementation inspired by the published model: five synthetic features pass through a two-dimensional bottleneck and are reconstructed with a ReLU output. It is real optimization in your browser, not a video or a measurement from a large language model.

Move sparsity, the chance that a feature is absent. When features rarely co-occur, overlap can become less costly. Directions can spread out so the model represents more features than the bottleneck has dimensions. With denser data, interference can make dropping some features worthwhile.

Watch the arrows and reconstruction loss. The displayed count uses a simple vector-length threshold; it does not by itself prove a particular polygon or recover a human concept. Shapes depend on the data, initialization, objective, and training settings.

The paper demonstrates a mechanism in controlled toy networks. It motivates questions about larger models without proving that every concept in every LLM has a clean linear direction. The little model has earned our curiosity; it has not earned a universal theory of everything.

In the LLM tracks you met attention as a mechanism — every token queries every earlier token. But mechanism isn’t purpose. What did training actually use attention to build? One well-studied example is: the induction head, a tiny algorithm for continuing patterns:

If you’ve seen [A][B] before, and you’re looking at [A] again — predict [B].

Once “Mr Dursley” has appeared, the next “Mr D” should continue “ursley”. No grammar, no meaning — pure pattern completion. The panel shows the attention map in the same rows-look-at-columns language as the attention scene. Read the bottom row: the head finds the previous occurrence of the current token and locks onto the token right after it — the teal-ringed cell. That target is what gets copied into the prediction.

Under the hood it’s a genuine two-part circuit: one head in an earlier layer marks each token with “what came before me,” and the induction head in a later layer uses that marking to find the match-and-look-ahead cell. Two learned parts, composing into one legible algorithm — discovered in the weights, not designed into them.

Now toggle trained off. That smear is what attention looks like before the circuit forms. In some studied small-model training runs, the transition occurs abruptly — a sudden drop in loss early in training (the “induction bump”) right when these heads snap into place. And their reach goes far beyond literal copying: induction heads are a leading candidate for why LLMs can do in-context learning — following patterns from your prompt that they were never trained on. Fuzzier versions match [A*][B*]…[A]→[B-ish], completing analogies rather than echoes.

The panel is a schematic, not a measured attention map. Evidence is strongest for the tested architectures and tasks; induction heads are not a complete explanation of every form of in-context learning. The next panels offer other ways to investigate learned representations.

A transformer updates internal representations across layers. The logit lens asks: what token scores do we get if we apply an output decoder to an intermediate representation?

That is a useful analytical trick. It is not listening to a thought mid-sentence. An intermediate layer wasn’t necessarily trained to use the final decoder in exactly that way, so the lens can distort what is present.

The panel’s two examples use authored numbers, arranged to illustrate a possible shift from generic continuations toward a particular answer. They are not measured GPT-2 outputs and do not demonstrate that every model follows a syntax → meaning → certainty staircase. Real trajectories can change course, remain uncertain, or settle on something wrong.

Choose a prompt and compare the rows. What changes? Which candidate leads? What would you need to know before concluding that a model “knew the answer” at a particular layer?

The tuned lens learns transformations to make intermediate decoding more reliable. Even a better lens remains an instrument with assumptions. Skipping layers safely or interpreting confidence requires additional testing; this illustration alone supports neither conclusion.

Sparse autoencoders can identify interpretable directions in a model’s activations. Researchers then test interventions: if we increase or decrease an activation, does the output change?

Anthropic’s Scaling Monosemanticity work found many interpretable features in Claude 3 Sonnet. Increasing a Golden Gate Bridge feature produced striking bridge-related responses, including text identifying the assistant as the bridge. That’s a result about generated behavior—not proof of a literal identity or experience.

Move the panel’s dial. The displayed outputs are written imitations of that kind of behavior. No language model is being steered here. The optional Neuronpedia button opens an external dashboard for a different, GPT-2 feature. It is not a live version of the same Claude experiment.

An intervention is stronger evidence than correlation alone: it can establish that a direction influences a behavior in the tested setting. It does not show that the feature is a complete, isolated concept or that effects will remain reliable across prompts and models.

Feature steering is promising for investigation and control, with unresolved issues around completeness, side effects, and robustness. There is no general “turn deception off” switch. The interesting question is what a carefully measured intervention can tell us, including when it fails.