Interpretability Needs the Interaction, Not the Feature

BAIR Berkeley's SPEX and ProxySPEX identify influential interactions in large language models at scale, where feature attribution sees only individual features. The Honest Architect reading is one level deeper: a feature is not a mechanism. The interaction is. On the trolley problem, SHAP blamed the word "trolley" and swapping it for synonyms did nothing; SPEX found the 4-word synergy (trolley×2 + pulling + lever), and swapping those four dropped the failure rate to near zero. The mechanism was the interaction, and the feature was a red herring.

Key takeaways

• SPEX (Spectral Explainer) and ProxySPEX identify influential interactions in LLMs at scale, extending interaction discovery from dozens to thousands of components (BAIR Blog, "Identifying Interactions at Scale for LLMs", 2026).
• A feature is not a mechanism. On the trolley problem, GPT-4o mini answered correctly only 8% of the time; SHAP blamed the word "trolley," but SPEX found the 4-word synergy (trolley×2 + pulling + lever), and replacing those four dropped the failure to near zero.
• ProxySPEX matches SPEX with ~10x fewer ablations by adding the hierarchy property to sparsity and low-degreeness.
• Theorem 3: a property is guaranteed exactly when its mechanism is implemented and measuring — faithfulness is the property, sparse recovery is the mechanism, and the interaction is what the mechanism measures.

SHAP found the feature; SPEX found the mechanism

In 2026, the BAIR Blog published "Identifying Interactions at Scale for LLMs," describing SPEX (ICML 2025) and ProxySPEX (NeurIPS 2025), algorithms that identify influential interactions in LLMs using ablation and sparse recovery. The cleanest demonstration is the trolley problem. A modified version where "True" is the clear correct answer still gets the correct answer from GPT-4o mini only 8% of the time. Standard feature attribution (SHAP) identified individual instances of the word "trolley" as the primary drivers of the incorrect response. Replacing "trolley" with synonyms like "tram" or "streetcar" had little impact — the feature was named, but the prediction did not change, because the feature was not the mechanism.

SPEX told a richer story. It identified a dominant high-order synergy between the two instances of "trolley" and the words "pulling" and "lever" — four words whose joint presence drove the failure, and whose individual presence did not. When those four words were replaced with synonyms, the model's failure rate dropped to near zero. That is the Honest Architect distinction in one example: a feature attribution tells you which feature correlates with the output; an interaction attribution tells you which mechanism produces it. SHAP measured a feature; SPEX measured the mechanism, and the mechanism was the interaction.

The implication for interpretability is the same one Theorem 3 draws for any property. A property is guaranteed exactly when its mechanism is implemented and measuring. "We know what drives the model" is a property, and it holds exactly when the measurement captures the interaction, not the feature. A feature attribution that misses the interaction is a measurement of the wrong thing, and a measurement of the wrong thing is worse than no measurement, because it makes the model look explained when it is not. The trolley case is the proof: SHAP said "trolley," the team swapped "trolley," and nothing changed, because the mechanism was four words acting together.

Sparsity, low-degreeness, hierarchy: the structure that makes recovery tractable

The reason SPEX works at scale is a structural observation, not a bigger computer. While the number of total interactions in a model grows exponentially with the number of features, data points, and components, the number of influential interactions is actually small. SPEX formalizes this through two properties: sparsity (relatively few interactions truly drive the output) and low-degreeness (influential interactions typically involve only a small subset of features). These turn an intractable search into a solvable sparse-recovery problem, borrowing tools from signal processing and coding theory — strategically selected ablations that combine many candidate interactions, then efficient decoding to disentangle them.

ProxySPEX adds a third property: hierarchy. Where a higher-order interaction is important, its lower-order subsets are likely to be important as well. That additional structure yields a dramatic improvement: ProxySPEX matches SPEX's performance with around 10x fewer ablations. On a problem where each ablation is an expensive inference call or a retraining, 10x fewer ablations is the difference between a method that runs in research time and one that runs in production time. SPEX extends interaction discovery from dozens to thousands of components, and ProxySPEX makes the thousands affordable.

The Honest Architect reading is that this is a measurement mechanism designed around the shape of the property. Sparsity, low-degreeness, and hierarchy are claims about how the influential-interaction property is structured, and the algorithm is built to measure exactly that structure. A mechanism that ignores the structure — exhaustive ablation, or marginal attribution that averages interactions away — measures the wrong thing at a cost the team cannot carry. The source names this: marginal approaches (LIME, Banzhaf) can operate at thousands of features but exhibit significantly lower faithfulness, because they fail to capture the complex interactions driving the output. Faithfulness is the property; the structure-aware mechanism is what holds it.

The Oracle measures the interaction, not the Sister

[PERSONAL EXPERIENCE] The HAI Engine has run in production since 2016, and the ensemble measurement we keep is the same shape as SPEX's interaction recovery. The Oracle normalizes probabilities in exactly one place, and the ensemble's sum-to-one and entropy are checked on every merge. The entropy is the interaction measurement: it tells us when the Sisters disagree enough to matter, and a low-entropy merge is a redundant interaction (the Sisters are saying the same thing) where a high-entropy merge is a synergistic one (the Sisters are contributing distinct evidence). That is the same distinction the source draws for data attribution — redundant interactions reinforce a concept, synergistic interactions define a decision boundary — and it is why we keep the entropy, not a per-Sister score.

A per-Sister measurement is the feature attribution of the ensemble. It tells you which Sister contributed which probability mass; it does not tell you which interaction between Sisters produced the calibrated forecast. The Oracle's merge is the mechanism, and the entropy is its measurement, in the same way that SPEX's sparse recovery is the mechanism and faithfulness is its measurement. We tag the Oracle's calibration as Production ✅ because the mechanism is implemented and the measurement is on every merge, not because we trust the Sisters individually. The Sisters are models; the merge is the property.

The source's data-attribution result is the same lesson at the training-data layer. ProxySPEX on a ResNet trained on CIFAR-10 found synergistic interactions where semantically distinct classes work together to define a decision boundary (an automobile sharing visual traits with a sports car, a truck, a delivery vehicle) and redundant interactions where visual duplicates reinforce a concept (a horse prediction influenced by a cluster of dog images with similar silhouettes). The fine-grained analysis lets you preserve necessary synergies while safely removing redundancies. The ensemble equivalent is: keep the high-entropy merges that carry distinct evidence; prune the low-entropy merges that carry duplicates. The entropy is the measurement that tells you which is which.

The topology is where the influential interactions live

[UNIQUE INSIGHT] The source's interaction discovery operates at three layers — feature, data, and model component — and misses a fourth that matters for any deployed system: the topology. The space is the router. A network, community, and room topology decides who sees what before anything responds, and that routing decision is an interaction — between the request, the room's scope, and the agent that fires. The influential interaction is not always inside the model; it is often between the model and the room the topology routed it to. An interpretability method that stops at the feature layer misses the routing interaction that decided which features the model ever saw.

This is the Honest Architect addition to the source's agenda. The source closes by naming the open question of unifying the three perspectives — feature, data, component — into a holistic understanding of a machine learning system. The unified view needs a fourth layer: the topology that routes the request to the model in the first place. A model that behaves differently in two rooms is not two models; it is one model whose influential interaction includes the room, and the topology is the mechanism that produces that interaction. Interpretability that ignores the topology will explain the model and miss the system.

The civil-and-defensive scope boundary is a topology-level interaction in our case. It is a written policy that decides which requests route to which agents, and the measurement is the deal we decline, observable in the pipeline. Customer sovereignty — your network, your brand, your data — is another: the routing rule that keeps the topology yours, with the export log as the measurement. Both are interactions between the request and the topology, and both are properties that hold exactly when their mechanism is implemented and measuring — the same test SPEX applies to features, applied one layer up.

Theorem 3: faithfulness is the property, sparse recovery is the mechanism

[ORIGINAL DATA] The 21-paper series specifies Theorem 3: a property is guaranteed exactly when its mechanism is implemented and measuring. Read it as the test for every interpretability claim. "We can explain the model's output" is a property, and it holds exactly when the measurement mechanism captures the influential interactions — not the features, not the marginal contributions, the interactions. SPEX's faithfulness score is the measurement; sparse recovery over sparsity, low-degreeness, and hierarchy is the mechanism; the property holds when the mechanism is implemented and the faithfulness score is on a dashboard someone looks at.

This is why our honesty tags are not adjectives. Production ✅ means the mechanism is implemented and its measurement is observed. The Oracle's entropy is Production ✅ because it is checked on every merge. The topology's routing is Production ✅ because the network, community, and room route before anything responds. An interpretability score is not a tag by itself; it is a measurement, and a measurement without a mechanism is a number, not a property. The source is honest about this — it evaluates SPEX by faithfulness, which is whether the recovered attributions predict the model's output on unseen test ablations, and that is the right test, because it measures whether the mechanism actually captures the property.

The same theorem is why we will not promise Wallet & Token, Super App, or Community Credit outcomes — those are Roadmap 🔵, the mechanism is not yet implemented and measuring, and a forecast we cannot measure is not a forecast we can honestly sell. Civil and defensive scope only, and no token or community-credit outcome claims, because the Howey review has not run on a mechanism that does not exist yet. To tag a Roadmap item with an interpretability result's glow would be the same error as calling a feature attribution a mechanism — a measurement of the wrong thing, presented as a property.

Frequently Asked Questions

What did SPEX find that SHAP missed on the trolley problem?

GPT-4o mini answered the modified trolley problem correctly only 8% of the time. SHAP identified individual instances of the word "trolley" as the drivers, but swapping "trolley" for synonyms did nothing. SPEX found a 4-word synergy (trolley×2 + pulling + lever), and replacing those four with synonyms dropped the failure rate to near zero. SHAP measured the feature; SPEX measured the mechanism, and the mechanism was the interaction.

Why does SPEX scale when exhaustive ablation does not?

Because of three structural properties: sparsity (few interactions truly drive the output), low-degreeness (influential interactions involve a small subset of features), and hierarchy (higher-order interactions imply their lower-order subsets matter). These turn an exponential search into a solvable sparse-recovery problem. ProxySPEX adds hierarchy and matches SPEX with ~10x fewer ablations.

How is the Oracle's entropy the same shape as SPEX's interaction recovery?

Both measure the interaction, not the individual. SPEX recovers influential interactions among features; the Oracle's entropy measures the interaction among Sisters on every merge. A low-entropy merge is a redundant interaction (duplicate evidence); a high-entropy merge is a synergistic one (distinct evidence). The per-Sister score is the feature attribution of the ensemble; the entropy is the mechanism measurement.

How does Theorem 3 apply to interpretability?

A property is guaranteed exactly when its mechanism is implemented and measuring. "We can explain the output" is a property that holds exactly when the measurement captures the influential interactions. SPEX's faithfulness is the measurement; sparse recovery is the mechanism. A feature attribution that misses the interaction is a measurement of the wrong thing, and a measurement of the wrong thing is worse than no measurement.

How does this map to Everythink's honesty tags?

Production ✅ means the mechanism is implemented and measuring — the Oracle's entropy on every merge, the topology's routing, the World Monitor's self-disable. An interpretability score is a measurement, not a tag. Roadmap 🔵 means the mechanism is not yet implemented, and no interpretability result upgrades it. The tags are the measurement of the mechanism, not a vibe about the model.

Sources

• BAIR Blog, "Identifying Interactions at Scale for LLMs" (SPEX / ProxySPEX), 2026, retrieved 2026-08-23, https://bair.berkeley.edu/blog/2026/03/13/spex/
• Tsai et al., "SPEX: Spectral Explainer", ICML 2025, referenced via BAIR Blog, https://openreview.net/forum?id=pRlKbAwczl
• ProxySPEX, NeurIPS 2025, referenced via BAIR Blog, https://openreview.net/forum?id=KI8qan2EA7

If your network is ready for measurement that captures the interaction, not the feature, create your network — the topology routes the request, the Sisters produce the drafts, and the Oracle measures the synergy on every merge.