Locally Coherent, Globally Incoherent: Bounding Compositional Incoherence in Multi-Component LLM Agents

Paper: 2605.30335 Authors: Anany Kotawala Categories: cs.AI, cs.CL

The Gap

Multi-component LLM systems (retrieval-augmented generation, agent ensembles, modular pipelines) are everywhere. Each component produces probabilistic outputs that are locally coherent—they satisfy probability axioms within their own scope. The assumption: if every part is coherent, the whole is coherent. Prior work on LLM calibration and ensemble methods focuses on improving individual component accuracy or aggregation rules, but assumes the composition inherits coherence from its parts.

This paper identifies a structural failure: when components see different slices of a joint problem and their outputs are composed, the result can violate basic probability axioms (e.g., probabilities summing to more than 1, contradictory conditional dependencies) even when every component is individually coherent. No existing framework measures or bounds this compositional incoherence at runtime.

Problem: Multi-component LLM agents
   |
   v
Assumption: Local coherence -> Global coherence
   |
   v
Reality: Composition violates probability axioms
   |
   v
Method: Formalize compositional residual eps*
        (L2 distance to coherent polytope)
   |
   v
Evidence: eps* > 0 on 33-94% of cliques
          +0.115 nats regret per bet
   |
   v
Conclusion: Local coherence does NOT suffice
            Intuitive mitigations FAIL

The Increment

One sentence: Before this paper, we assumed multi-component LLM systems inherit coherence from their parts; after, we know they systematically violate probability axioms and can quantify the violation at runtime.

Core Mechanism

The method has three layers. First, formalization: define the compositional residual eps** as the L2 distance from the composed probability distribution (what the system outputs) to the nearest point in the coherent polytope (the set of all distributions satisfying probability axioms given the declared cross-component constraints). This is computable at runtime from system output alone.

Second, characterization: establish a product-structure dichotomy. When components are independent (product structure), local coherence guarantees global coherence (eps** = 0). When components share variables (coupled structure), local coherence does NOT suffice. A Rayleigh-quotient predictor estimates eps* from the coupling structure before running the system, matching observed residuals within 7% on three of four relation classes.

Third, repair and monitoring: a hierarchical Boyle-Dykstra projection algorithm takes the incoherent composition and projects it onto the coherent polytope deterministically. An anytime-valid e-process provides sequential coherence monitoring—detecting when the system drifts into incoherence during a multi-turn interaction.

Input: Component outputs (local coherent distributions)
  |
  v
[Compose] --> Composed distribution Q
  |
  v
[Measure] --> eps* = distance(Q, coherent polytope)
  |
  v
[Predict] --> Rayleigh quotient estimates eps* from structure
  |
  v
[Repair] --> Boyle-Dykstra projection --> Q' (coherent)
  |
  v
[Monitor] --> e-process tracks coherence over time

Think of it like building a bridge from modular sections. Each section (component) is structurally sound when tested in isolation—bolts tight, welds clean, load-bearing capacity verified. You assume the assembled bridge inherits this soundness. But when you bolt the sections together, the joints don’t align perfectly. The cumulative misalignment creates stress concentrations that violate the bridge’s load-bearing assumptions, even though every section passed inspection.

The compositional residual eps* is the total misalignment at the joints. The product-structure dichotomy says: if sections are independent (no shared load paths), misalignment doesn’t propagate. If sections share load (coupled structure), misalignment compounds. The Rayleigh quotient predicts misalignment from the blueprint before assembly. The Boyle-Dykstra projection is the hydraulic jack that forces the joints into alignment. The e-process is the strain gauge that monitors alignment during operation.

Key Concepts

• Coherent polytope: Imagine all possible probability distributions over a set of events as points in a high-dimensional space. Not all points are valid—some violate basic rules like “probabilities must sum to 1” or “if A implies B, then P(A) ≤ P(B).” The coherent polytope is the subset of points that satisfy all probability axioms and the declared constraints between components. It’s a convex region: if two distributions are coherent, any weighted average of them is also coherent. The compositional residual eps** measures how far outside this region the composed distribution lands. Concrete example: Component A says P(rain) = 0.6, Component B says P(rain AND cold) = 0.5. If they’re composed naively, you might output both claims. But P(rain AND cold) cannot exceed P(rain), so the composition is outside the coherent polytope. The distance to the nearest valid point (e.g., adjusting to P(rain) = 0.6, P(rain AND cold) = 0.5) is eps*.

• Product-structure dichotomy: This is the paper’s central theoretical result. If components operate on disjoint variable sets (product structure), their joint distribution is the product of their marginals, and local coherence guarantees global coherence. If components share variables (coupled structure), the joint distribution must satisfy cross-component constraints (e.g., marginalization, conditional independence), and local coherence does NOT guarantee global coherence. Example: Two weather forecasters, one predicting rain (yes/no), one predicting temperature (hot/cold). If they’re independent (product structure), their joint forecast is automatically coherent. If they share a variable (e.g., both predict “rain” but use different definitions), their forecasts can contradict even if each is individually coherent.

• Rayleigh quotient predictor: Before running the system, you can estimate eps** from the coupling structure alone. The Rayleigh quotient is a ratio: (variance of constraint violations) / (total variance). It captures how much of the system’s uncertainty is concentrated in the cross-component constraints versus within-component uncertainty. High Rayleigh quotient → high predicted eps*. The paper shows this predictor matches observed eps* within 7% on three of four relation classes, meaning you can anticipate compositional incoherence from the system architecture before deployment.

Framework Shift

Before (mainstream approach):        After (this paper):

Component A --> [Aggregate] --> Output    Component A --+
Component B ----^                         Component B --|--[Compose]--> Q
Component C ----^                         Component C --+       |
                                                                v
Assumption: If A, B, C coherent,                      [Measure eps*]
            output coherent                                     |
                                                                v
Focus: Improve A, B, C individually              eps* > 0? --> [Repair via projection]
       or aggregation rule                                     |
                                                                v
                                                       Q' (coherent output)
                                                                +
                                                       [Monitor via e-process]

One sentence: From assuming composition inherits coherence to measuring and repairing compositional incoherence as a first-class system property.

Expert Assessment

Problem choice: Real gap. Multi-component LLM systems are production reality (RAG, agent frameworks, ensemble forecasting), and the assumption that local coherence suffices is implicit everywhere. The paper makes the invisible visible. It sits at the intersection of probabilistic reasoning and LLM system design—a natural next question after calibration research.

Method maturity: The formalization (eps** as L2 distance to coherent polytope) is clean and computable. The product-structure dichotomy is a genuine theoretical contribution. The Rayleigh quotient predictor is clever—predicting failure from structure alone is valuable. The Boyle-Dykstra projection is borrowed from convex optimization, not novel, but appropriately applied. The negative results (three intuitive mitigations all fail) are as important as the positive results. No simpler approach is being overlooked; the problem requires this level of machinery.

Experimental integrity: The setup is honest. Four mid-tier LLMs, 1,876 ensemble cliques, 1,770 resolved bets. The baselines (retrieval, partition-aware prompting, aggregator-LLM) are the obvious things practitioners would try, and the paper shows they don’t work. The +0.115 nats regret is meaningful in forecasting contexts. One concern: the gain collapses to +0.006 when bettors themselves coherentize, suggesting the problem might be less severe in adversarial settings where downstream users apply their own coherence checks. The paper acknowledges this but doesn’t explore it deeply. The frontier-panel rerun (Section 5.5) is mentioned but not detailed in the abstract, which is a writing choice that hides potentially important results.

Writing quality: The abstract is dense and assumes familiarity with coherence theory and betting markets. The paper would benefit from a concrete running example in the introduction—show a real multi-component LLM system producing an incoherent output, compute eps**, apply the repair, and show the before/after. The formalization is rigorous but could use more intuition-building before diving into polytopes and Rayleigh quotients. The negative results section (mitigations that fail) is the most valuable part and should be elevated.

Verdict: weak accept — Identifies a real, underexplored failure mode in production LLM systems and provides both theory and runtime tools to address it, but the writing assumes too much background and the practical impact depends on deployment context.

Takeaways

For practitioners building multi-component LLM systems: Don’t assume your ensemble is coherent just because each component is calibrated. Compute eps** at runtime (the paper provides the formula). If eps* > 0, apply the Boyle-Dykstra projection before serving the output. This is especially critical for forecasting, decision support, or any application where probabilistic outputs are consumed by downstream reasoning.

For researchers: The product-structure dichotomy is a transferable insight. Any modular probabilistic system (not just LLMs) can exhibit this failure. The Rayleigh quotient predictor is a general tool for anticipating compositional failures from architecture alone—useful for system design and ablation studies.

For system designers: The three failed mitigations (retrieval, partition-aware prompting, aggregator-LLM) are instructive. Giving components more context or adding a meta-component doesn’t fix the structural problem. The failure is in the composition operator, not the components. Design your aggregation layer to enforce coherence constraints explicitly, not implicitly.

Steal this: The e-process for sequential coherence monitoring. If you’re building a multi-turn agent system, this gives you a principled way to detect when the agent’s beliefs are drifting into incoherence over time, which is a leading indicator of downstream failures.