LLMs Know the Parts But Fail at the Whole: Statistical Self-Consistency as a Blind Spot

Paper: 2607.15277 Authors: Patrik Wolf, Thomas Kleine Buening, Andreas Krause, Celestine Mendler-Dünner Categories: cs.CL

The Gap

People have been treating in-context learning as conditional inference for a while now: you describe a subpopulation, the model gives you a distribution, and you hope it’s a reasonable estimate. Benchmarks test accuracy against ground truth, calibration against expected frequencies, and prompt sensitivity against paraphrases. But nobody has seriously asked: do LLM estimates obey the basic axioms that any probability estimator must satisfy?

Specifically, if you ask a model “what fraction of Americans support X?” and also ask about men, women, young, old, etc., then the law of total probability says you should be able to reconstruct the population answer from the subpopulation answers weighted by subpopulation sizes. This is a reference-free consistency check — you don’t need ground truth to detect failure.

The gap: existing evaluation either needs ground truth (accuracy metrics) or tests surface-level properties (calibration). Nobody has tested whether LLM outputs satisfy the compositional structure that the conditional-inference interpretation demands. If they don’t, the whole framing of “LLMs as approximate Bayesians” is on shaky ground.

LLMs as conditional estimators
         |
         v
This interpretation predicts
probabilistic consistency
         |
         v
Partition population into subgroups
         |
         v
Prompt for each subgroup's distribution
         |
         v
Aggregate via law of total probability
         |
         v
Compare aggregated vs. direct estimate
         |
         v
Widespread violations found
         |
         v
Models have subgroup knowledge
but fail at aggregation
         |
         v
"Macro fallacy": fine-grained > coarse

The Increment

One sentence: Before this paper, nobody systematically tested whether LLM probability estimates obey the law of total probability; after it, we know they don’t — and that the violations are *informative, revealing that models know more at the subpopulation level than they let on at the population level.

Core Mechanism

The method has three clean phases — partition, prompt, aggregate — and one crucial diagnostic loop.

Phase 1: Partition. Take any population (e.g., “US adults”) and recursively split it using a binary tree. Level 0 is the whole population. Level 1 might split by gender. Level 2 splits each gender cell by age. And so on. Each leaf node is a subpopulation described in natural language.

Phase 2: Prompt. For each node in the tree (including internal nodes), verbalize the subpopulation description as a prompt: “What fraction of [subgroup] would answer yes to [question]?” Record the model’s output probability. Do this for every node at every level.

Phase 3: Aggregate. Using the law of total probability, combine leaf-level estimates back up the tree. At each internal node, the aggregated estimate equals the weighted sum of its children’s estimates, weighted by the known (or estimated) subpopulation sizes. Compare the aggregated estimate at each level with the direct prompt estimate at that same level.

The gap between the direct estimate and the aggregated estimate is the consistency violation. The authors measure this across multiple domains (opinion polling, demographic questions), multiple frontier models, and multiple tree structures.

                  [Level 0: "All US adults"]
                  /                         \
      [Level 1a: Men]              [Level 1b: Women]
       /            \               /             \
  [Young M]    [Old M]        [Young W]      [Old W]

For each node: prompt -> get p_est
For each internal node:
  p_agg = w_left * p_left + w_right * p_right
  violation = |p_est - p_agg|

Structural metaphor: The Census Bureau That Can’t Add.

Imagine a Census Bureau that sends surveyors to every county. Each county surveyor returns an estimate: “In our county, 62% support policy X.” The Bureau then needs to compute the state-level number by weighting each county’s answer by its population. Simple arithmetic.

Now imagine the Bureau also independently surveys the state as a whole and gets a different number — say 55%. That’s a problem: the direct survey and the aggregated survey disagree. Someone’s math is wrong, or the surveyors are systematically biased in ways that don’t compose.

This paper does exactly this with LLMs. Each “county surveyor” is a prompt conditioned on a subpopulation description. The “state-level survey” is the direct population prompt. The “weighting” is the law of total probability. And the consistent finding is: the Bureau can’t add. The county-level estimates, when properly weighted, don’t match the state-level estimate. And here’s the twist — the county-level numbers are often more accurate against real-world polling data than the state-level number. The Bureau’s local offices know more than headquarters.

The method’s components map directly: tree nodes = counties, prompt = survey question, aggregation = population weighting, consistency violation = the discrepancy between local and central reporting.

Key Concepts

• Law of Total Probability: Imagine you have a bag of marbles — 60% red, 40% blue. Among red marbles, 30% have a dot. Among blue marbles, 70% have a dot. The overall fraction of dotted marbles is 0.6 × 0.3 + 0.4 × 0.7 = 0.46. This is the most basic rule of how parts compose into a whole. If someone gave you answers for the red and blue groups that didn’t add up to the total, you’d know something is broken. That’s exactly what this paper tests, with “marbles” = people and “dotted” = “would say yes.”

• The Macro Fallacy: This is the paper’s key discovery. When you ask an LLM “what do Americans think about X?”, it gives you a mediocre answer. But when you ask “what do young urban Americans think?” and “what do rural retired Americans think?” separately and combine, you get a *better answer — one closer to real polling data. The model is like a doctor who gives vague general advice but excellent specific advice when you tell them exactly who the patient is. The “macro” (population-level) estimate suffers from some kind of averaging-out blur, while the “micro” (subpopulation) estimates are sharper and more grounded.

• Reference-Free Evaluation: Most AI benchmarks need a “right answer” to compare against. Self-consistency is different: it’s like checking whether someone’s arithmetic is internally consistent without knowing what the correct sum is. If A + B should equal C, and it doesn’t, you’ve found a bug — even without knowing what C *should be. This makes the evaluation framework reusable across any domain where you can define a meaningful partition.

Framework Shift

Before (mainstream approach):        After (this paper):

  Prompt LLM                         Partition population
       |                                    |
       v                                    v
  Get answer                         Prompt per subgroup
       |                                    |
       v                                    v
  Compare to ground truth            Aggregate via probability
                                           |
                                           v
                                    Compare aggregated
                                    vs. direct estimate
                                           |
                                           v
                                    No ground truth needed;
                                    consistency IS the test

From testing accuracy against external truth to testing internal coherence against probabilistic axioms, the core shift is treating LLMs as probability estimators that must satisfy structural constraints, not just point-estimate machines that need to be close to a number.

Expert Assessment

Problem choice: This is a genuine, clean gap. The conditional-inference framing of in-context learning is widespread (Garg et al., Xie et al., Abernethy et al.), but nobody followed through on the obvious implication: if it’s really inference, it should satisfy the axioms. The paper picks one axiom — compositionality via the law of total probability — and tests it rigorously. That’s elegant. The problem sits right at the intersection of probabilistic reasoning and prompt engineering, which is a live and underexplored area.

Method maturity: The binary-tree partition scaffold is clever but not overengineered. It gives structure to what could be an ad hoc exploration. The macro fallacy finding is the real contribution — it’s surprising, reproducible, and practically useful. One weakness: the method assumes you know or can estimate subpopulation sizes (priors), and the paper doesn’t deeply explore sensitivity to misspecified priors. Also, there’s no attempt to *fix the inconsistency — the paper is diagnostic, not prescriptive.

Experimental integrity: The experiments span multiple domains (GSS survey questions, demographic splits), multiple frontier models, and multiple tree structures. That’s solid. The macro fallacy effect is shown to persist across variations, which is convincing. One concern: the choice of partition tree (what to split by first, how deep to go) is still somewhat arbitrary, and the paper acknowledges but doesn’t fully resolve whether the results are tree-structure-dependent. The comparison against human reference data (GSS ground truth) is a nice validation that the subpopulation estimates are actually better, not just different.

Writing quality: The paper is well-written for its audience. The three-phase structure (partition, prompt, aggregate) is easy to follow. The main weakness is Section 5 on “implicit prompting” — it’s brief and somewhat hand-wavy. The authors show you can partially recover consistency by hinting at the aggregation structure in the prompt, but the treatment feels like an afterthought. If Section 5 were expanded into a full ablation on *why the macro fallacy occurs and what prompt structures mitigate it, the paper would go from “interesting finding” to “actionable insight.”

Verdict: weak accept — clean problem, clean method, surprising finding, but the diagnostic-only nature and underexplored mechanistic explanation keep it from strong accept territory.

Takeaways

1. Don’t trust a single LLM call for population-level estimates. If you’re using LLMs to estimate demographic or subgroup-level properties, break the population into meaningful subgroups, estimate separately, and aggregate yourself. The macro fallacy suggests this consistently outperforms direct estimation.

2. Self-consistency as a debugging tool. Whenever you use an LLM to estimate something compositional (market segments, risk categories, sentiment across groups), check whether the parts sum to the whole. If they don’t, trust the parts more than the whole.

3. Subpopulation prompting is a cheap accuracy boost. You don’t need new models or fine-tuning. Just condition on relevant demographic/contextual descriptors when prompting, then weight-average the results. This paper gives a principled justification for what good prompt engineers already do intuitively.