Beyond a Bag of Features: When SAE Latent Sets Stop Behaving Like Sets

Paper: 2608.11197 Authors: Nikolai Bolik, Lennart Stöpler, Artur Andrzejak Categories: cs.LG, cs.CL

The Gap

Shani et al. (2026) asked a clean question: do LLM representations carve up the world the way humans do? Their answer was half-yes. Using cosine similarity over dense embeddings, they found models broadly recover human category boundaries (a robin and a penguin are both birds, and neither is furniture) but miss typicality structure (humans strongly feel a robin is a better bird than a penguin; cosine similarity mostly shrugs).

The natural objection is that cosine similarity is a blunt instrument. It collapses a whole high-dimensional state into one scalar and gives you no handle on why two things are close. The interpretability community’s answer to exactly this complaint is sparse autoencoders: decompose the residual stream into a large dictionary of mostly-inactive, individually-nameable latents. Under that view, a concept is not a direction but a set of active features, and similarity becomes something you can read off — how many named features do these two things share?

That reframing carries a hidden assumption, and this paper’s contribution is to name it and test it. If concepts are sets of features, then composition should be roughly union-like: “red car” should activate approximately the features of “red” plus the features of “car.” Call this bag-of-features semantics. Nobody in the SAE literature quite claims this out loud, but a great deal of practice — feature-list explanations, “this neuron fires for X” write-ups, set-overlap probing — quietly depends on it.

[Shani et al. 2026]  dense cosine sim
   categories: recovered ~ typicality: missing
              |
              v
  (Assumption under test) sparse latents are named units,
      so concept == SET of active latents, and
      composition == UNION of sets
              |
              v
  [Method] SAE active latent sets + set-overlap similarity
              |
    +---------+-----------+--------------------+
    v                     v                    v
[E1: toy models]   [E2: human concepts]  [E3: controlled edits]
 union structure     no improvement        human-judged change
 IS recovered        over dense / resid    != set change
    |                     |                    |
    +---------+-----------+--------------------+
              v
  [Conclusion] set overlap tracks model-internal
  structure, not human structure; bag-of-features
  semantics holds only in idealised settings

The Increment

One sentence: Before, “which SAE features are active” was treated as an interpretable stand-in for “what the model means here”; after, we have evidence that the *set of active latents composes non-unionly and aligns with model-internal similarity rather than human conceptual similarity — so feature lists are not a semantics.

Core Mechanism

The machinery is deliberately simple, which is the right call for a claim like this. Take a token or text, run the LLM, grab the residual stream at some layer, push it through a trained SAE encoder. You get a sparse activation vector; throw away the magnitudes and keep the support — the identities of the latents that fired. That’s your set. Similarity between two inputs becomes a set-overlap score (Jaccard-style: shared latents over total latents). Everything downstream is a comparison of this measure against dense cosine similarity and raw residual-stream cosine similarity.

The paper’s structure is three experiments in increasing adversariality. First, a validation stage: in controlled toy models where the ground-truth generative structure is known to be compositional, do SAE latent sets actually recover union-like composition? Yes. And on natural text, do high-overlap neighborhoods look semantically coherent to a human eye? Also yes. This matters — without it, any later negative result could be dismissed as “your similarity measure was broken.”

Second, the replication-and-extension: rerun the human-concepts analysis with set overlap in place of cosine. If sparse features are the more human-aligned representational format, set overlap should recover category boundaries better and — the real prize — start picking up typicality gradients. It does neither. It tracks how the model organizes things.

Third, the diagnostic that gives the paper its title. Take an input and apply a well-controlled semantic modification — a change whose conceptual magnitude humans can rate reliably. Then measure how much the active latent set moved. If bag-of-features semantics held, small conceptual edits would swap a few features and large ones would swap many. Instead there’s substantial mismatch: the set can churn under a semantically trivial edit, or barely budge under a large one.

   input x                          input x'  (controlled edit)
      |                                 |
      v                                 v
  [ LLM layer L ]                  [ LLM layer L ]
      |  h                             |  h'
      +--> dense cos(h,h')  ......... baseline 1
      |
      v
  [ SAE encoder ]                  [ SAE encoder ]
      |  a (sparse)                    |  a'
      v                                v
   S = supp(a)                     S' = supp(a')
   (drop magnitudes)
      |                                |
      +--------------+-----------------+
                     v
        overlap(S,S') = |S n S'| / |S u S'|
                     |
      +--------------+--------------+
      v              v              v
 toy: union     human concepts:  edits: overlap
 recovered      no gain vs cos   drift != human
                                 rated change

Here’s the load-bearing analogy. Think of the SAE as a scanner that reads a dish and prints an ingredient list. The active set is the list; overlap is “what fraction of ingredients do these two dishes share.” In a salad bar — the toy model — this works perfectly: a Greek salad is the union of its ingredients, adding feta changes exactly one line on the list, and two salads sharing eight of ten ingredients really are similar dishes. That’s the validation experiment.

Now take the scanner into a real kitchen. A cake and its batter share nearly every ingredient and are not the same dish; a teaspoon of salt versus a tablespoon changes no line on the list but ruins the thing; browning the butter transforms the flavor while the receipt is identical. That’s the third experiment: the ingredient list is real, the scanner works, but the list is not the dish, because cooking is not union. And when you sort dishes by receipt overlap, you get a taxonomy that reflects the pantry’s organization — what the supplier stocks together — rather than how eaters group meals. That’s the second experiment: set overlap tracks model-internal structure.

Key Concepts

• Active latent set: An SAE takes the model’s internal state at a point in the text and rewrites it as a mostly-zero list of thousands of numbers, where each nonzero slot is supposed to mean something nameable (“legal language,” “the letter Q,” “something is being negated”). If you ignore *how strongly each slot fired and just note which ones fired, you get a set — like a checklist of ticked boxes. Two inputs are then “similar” if their checklists overlap. The appeal is that you can read the checklist; the risk, which this paper exploits, is that ticking a box discards how hard it was ticked.

• Typicality structure: Humans don’t experience categories as yes/no. Ask people to rate how good an example of “bird” each animal is and you get a stable gradient: robin above eagle above ostrich above penguin. Same category, graded membership. This gradient predicts real behavior — reaction times, what people list first, what children learn first. A representation that gets boundaries right but the gradient wrong knows *bird vs chair but not what birdiness feels like from the inside.

• Bag-of-features semantics: The assumption that meaning is a set and combination is set union — that the features for “black cat” are the features for “black” plus the features for “cat,” nothing added, nothing lost. It’s the same move as bag-of-words in old-school NLP: throw away arrangement and interaction, keep presence. It’s tempting because it makes explanations trivially compositional, and it’s exactly what fails here outside idealized settings. Interaction terms and magnitudes are doing semantic work that set membership can’t see.

Framework Shift

Before (mainstream SAE practice):     After (this paper):

  h  in  R^d                            h  in  R^d
   |                                      |
   v                                      v
[ SAE ] --> \{f3, f17, f402\}           [ SAE ] --> \{f3, f17, f402\}
   |                                      |
   v                                      |  set is REAL but
"meaning = this list"                     |  not a semantics
   |                                      v
   v                              small human edit --> big set churn
compose by UNION:                 big human edit   --> tiny set churn
 \{red\} + \{car\} = \{red, car\}          |
   |                                      v
   v                              overlap ~ model-internal
explain, probe, edit              geometry, NOT human concepts
via feature lists
                                  union holds only in toy /
                                  ground-truth-compositional regimes

From geometry to combinatorics and back again: the core shift is that making similarity readable does not make it human-aligned, and set membership is the wrong abstraction level because it throws away the magnitudes and interactions where composition actually lives.

Expert Assessment

Problem choice: Real gap, well positioned. The field has spent two years scaling SAEs and building tooling on top of feature lists, with the compositional assumption mostly implicit. Papers that name a load-bearing assumption and stress-test it are worth more than another dictionary-scaling result. Riding on a very recent paper (Shani et al. 2026) is a double-edged move: it gives a ready-made benchmark and a clear comparison, but it also means the framing has a short shelf life and inherits whatever flaws the original human-concepts setup has.

Method maturity: Clever framing, deliberately unclever machinery — appropriate for a negative result, since a fancy method would just invite “your method was broken.” The toy-model and neighborhood-coherence validation before the negative claims is exactly the right discipline; skipping it is the most common way this genre of paper fails. But the paper is testing a strawman-adjacent version of the assumption, and the obvious next step is conspicuous: run the same analyses with *magnitude-weighted SAE activations. If weighted cosine over SAE activations recovers what binary set overlap loses, then the finding is “don’t binarize” rather than “features don’t compose” — a much narrower claim. From the abstract I can’t tell whether they ran this; if they didn’t, it’s the first thing a reviewer should demand. Likewise, the “controlled semantic modifications” experiment is the strongest part and deserves to be the paper’s center of gravity rather than a probe at the end.

Experimental integrity: The baseline choice is fair and non-trivial — comparing against both dense embeddings *and raw residual-stream states forecloses the easy objection that the residual stream itself was the problem. The structural risk in all SAE negative results is generalization across the SAE design space: layer choice, dictionary width, sparsity coefficient, and architecture (TopK vs JumpReLU vs gated) all change the active set substantially, and known pathologies like feature absorption and feature splitting will corrupt set overlap in ways that have nothing to do with composition. A finding that holds for one SAE at one layer of one model is a finding about that SAE. Also worth flagging: human typicality datasets are small and noisy, so “no improvement over dense” needs to be a real gap rather than a difference inside the noise floor. Note that I’m assessing from the abstract, not the full experimental section — these are the questions to check, not confirmed defects.

Writing quality: The abstract is unusually honest and well-sequenced — validate the measure, then break it — and it resists overclaiming (“outside idealised settings” is the right hedge, and rarer than it should be). The likely corner-cutting is in the toy-model section, which is easy to write as “we constructed a case where it works” without characterizing *where the boundary is between the toy regime and the real one. That’s the rewrite that would elevate the paper: don’t just show union-composition holds in toy models and fails in text — find the knob (interaction strength? feature co-occurrence density? sparsity level?) that carries you from one regime to the other. That turns a negative result into a predictive one.

Verdict: weak accept — a well-scoped, properly-validated negative result against an assumption the field genuinely relies on, held back by unclear robustness across SAE configurations and an unaddressed magnitude-weighted alternative that could substantially narrow the claim.

Takeaways

Concrete things to steal:

• Validate your interpretability measure before you use it to draw conclusions. The two-step of “recover known structure in a toy model with ground truth, then check neighborhoods are semantically coherent in the wild” is a reusable template. Any paper claiming X *doesn’t have property P needs this, or the result is unfalsifiable.
• The controlled-edit protocol transfers directly. Apply a semantic modification whose magnitude humans can rate, then measure representation drift, then correlate. This is a cheap, general test for “does my representation’s notion of distance match anyone’s,” and it works for embeddings, probes, attention patterns, or activation steering — not just SAEs.
• Binarizing is a modeling choice with teeth. Whenever you convert a continuous representation into “which units are on,” you have assumed that magnitude carries no semantics. Sometimes that’s fine. Here it isn’t, and the paper is a good reminder to ablate the binarization rather than inherit it from convention.
• Practical caution if you build on SAEs: don’t ship pipelines that assume feature-set union is composition — feature-list diffs, set-overlap retrieval, “concept = these N latents” dashboards. They will look sensible on curated examples and drift incoherently on real edits. Prefer weighted comparisons and validate against human-rated change on your own data.
• Framing to keep: interpretability and alignment-with-humans are independent axes. Making a similarity measure *readable tells you nothing about whether it measures what humans mean. Easy to conflate; expensive to conflate.