Operadic Consistency: Catching LLM Reasoning Failures Without Labels

Paper: 2606.13649 Authors: Nathaniel Bottman, Yinhong Liu, Kyle Richardson Categories: cs.CL, cs.LG

The Gap

Existing methods for detecting LLM reasoning failures without ground-truth labels fall into two camps: (1) sampling-based approaches like self-consistency (CoT-SC) and semantic entropy, which rely on multiple independent outputs from the same question, and (2) self-evaluation approaches like P(True), where the model judges its own confidence. Both treat the model as a black box and ignore the structure of the reasoning—specifically, whether the model’s answer is consistent with its own decomposition of the question.

The paper identifies a blind spot: for compositional (multi-hop) questions, a model should produce the same answer whether it answers directly or first breaks the question into sub-questions, answers each, then composes the results. No prior work formalized this as a diagnostic. The authors fill this gap by borrowing operad theory (a formalism for systems built by iterated substitution) and grounding a simple comparison: direct answer vs. composed answer.

        +------------------------------+
        | Problem: No label-free signal|
        | for compositional failure   |
        +-------------+---------------+
                      |
                      v
        +-------------+---------------+
        | Assumption: For composition |
        | queries, direct answer      |
        | should equal composed       |
        | answer from decomposition   |
        +-------------+---------------+
                      |
                      v
        +-------------+---------------+
        | Method: Compute OC =        |
        | agreement( direct answer,   |
        | compose( decompose( Q ) ) ) |
        +-------------+---------------+
                      |
                      v
        +-------------+---------------+
        | Evidence: OC correlates     |
        | with accuracy (r 0.86-0.94) |
        | across 12 models, 4 datasets|
        +-------------+---------------+
                      |
                      v
        +-------------+---------------+
        | Conclusion: OC is a         |
        | label-free signal for       |
        | compositional failure       |
        +------------------------------+

The Increment

One sentence: Before this paper, reasoning failure detection relied on output statistics or self-evaluation; after this paper, we gain a structural consistency check that exploits the model’s own decomposition of the query, providing complementary signal at little extra cost.

Core Mechanism

Operadic consistency works in three steps:

1. Extract decomposition: Given a multi-hop question Q, prompt the model to produce a decomposition into sub-questions (e.g., via chain-of-thought or a separate prompt). Also record the model’s direct answer A_direct.

2. Compose sub-answers: For each sub-question, obtain the model’s answer (using the same model or a copy). Then define a composition function that combines the sub-answers into a full answer A_composed. The composition function depends on the decomposition structure—e.g., if the sub-questions are linear, answers are simply concatenated or a final answer is derived.

3. Compare: Compute the agreement (e.g., exact match or a soft score based on lexical overlap or probability) between A_direct and A_composed. This agreement is the operadic consistency score—higher agreement signals higher confidence that the model’s reasoning is correct.

The whole pipeline is inference-only, requires no training, and for the K-shot budget (K=3) adds negligible latency.

                           +---------------------+
                           | Input Question Q    |
                           +----------+----------+
                                      |
                  +-------------------+-------------------+
                  |                                       |
                  v                                       v
     +------------+------------+             +------------+------------+
     | Decompose Q into        |             | Direct answer:          |
     | sub-questions q1..qn    |             | A_direct = model(Q)    |
     +------------+------------+             +------------+------------+
                  |                                       |
                  v                                       |
     +------------+------------+                           |
     | Answer each sub-q       |                           |
     | -> a1 = model(q1), etc |                           |
     +------------+------------+                           |
                  |                                       |
                  v                                       |
     +------------+------------+                           |
     | Compose answers:        |                           |
     | A_composed = compose(   |                           |
     |   a1..an )              |                           |
     +------------+------------+                           |
                  |                                       |
                  +-------+-------------------------------+
                          |
                          v
                +---------+---------+
                | Compare A_direct  |
                | vs A_composed     |
                | -> OC score       |
                +-------------------+

Structural metaphor: the jigsaw puzzle check.
Imagine you have an assembled jigsaw puzzle (the question Q). You can either glance at the full picture and announce “it’s a castle” (direct answer), or you can follow an instruction sheet that tells you to separate the puzzle into four corner pieces, examine each, then mentally reassemble them to decide “it’s a castle” (decomposition + composition). If your first answer and your assembled answer match, you likely understood the picture; if they contradict, something went wrong—maybe you misread a corner piece, or the sheet was wrong, or you hallucinated a dragon. The operadic consistency score is the “match signal”. Each component maps: the instruction sheet = decomposition prompt; examining each corner = answering sub-questions; mental reassembly = composition; final verbal claim = direct answer.

Key Concepts

• Operadic consistency (OC)
  Operad theory studies systems where complex structures are built by plugging simpler pieces into slots. In language, a multi-hop question is an operad: the top-level query has “slots” for sub-question answers. OC checks that the model respects this plugging operation—i.e., that the output of the full query equals the result of inserting sub-answers into the slots. It’s a formal way of saying “the model’s answers are self-consistent under decomposition”.

• Multi-hop QA
  Questions that require reasoning across multiple facts, e.g., “What is the capital of the country where the Eiffel Tower is located?” The first hop locates France, the second hop finds Paris. Multi-hop datasets (HotpotQA, MuSiQue, StrategyQA) are a natural testbed because decomposition is explicit and composition is well-defined.

• Self-consistency (CoT-SC)
  A baseline that samples multiple reasoning paths and aggregates by majority vote. It detects failures when different paths converge to different answers. OC differs: it uses a single path (the decomposition) and checks internal consistency, not aggregation of multiple paths. The two signals are complementary—OC captures structural errors that CoT-SC might miss when all paths hallucinate the same wrong answer.

Framework Shift

The core shift is from aggregating distributional statistics to checking structural agreement.

Before (mainstream approach):
   Q --> [LLM x K] --> {answers} --> majority vote / entropy --> confidence
   The decomposition structure is ignored; only output patterns matter.

After (this paper):
   Q --> [LLM] --> A_direct
   Q --> [LLM] --> decomposition --> sub-answers --> compose --> A_composed
         OC = match(A_direct, A_composed)
   Structure is front-and-center: consistency under plugging reveals failures.

From output statistics to structural consistency: the old view treats each answer as an independent draw; the new view exploits the fact that reasoning has internal dependencies that should be invariant under decomposition.

Expert Assessment

Problem choice: Real gap. Researchers knew self-consistency and semantic entropy had blind spots (e.g., when all sampled paths are wrong), but no one formalized the decomposition-as-consistency-check. This sits squarely in the “confidence calibration for reasoning” trajectory and is timely given the proliferation of multi-hop QA evaluations.

Method maturity: Clever but simple. The authors resisted overcomplicating—they directly compare answers, no learned model, no extra training. The elegance is the insight, not the algorithm. However, a simpler baseline (e.g., checking if the final answer appears in the CoT) would have been nice to see; the paper’s strength is in the controlled comparisons they do provide.

Experimental integrity: Solid. 12 models from 4B to 671B, 4 datasets, multiple baselines (CoT-SC, semantic entropy, P(True)). They report cluster-robust p-values, controlling for multiple comparisons. The equal-cost comparison (K=3) is fair because they match inference budget. No obvious red flags—results are consistent and effect sizes are large. The only minor concern: the decomposition extraction procedure uses the model’s own CoT, which introduces a confound (the decomposition may be poor), but this is addressed in ablation (they also use human-written decompositions).

Writing quality: The abstract and introduction are clear. The paper could benefit from a standalone example walkthrough—the operad formalism might scare some readers. The “Related Work” section is terse but adequate. If I could rewrite one part, it would be Section 3 (Method): add a fully worked example with actual questions and answers to make the pipeline concrete.

Verdict: strong accept — a novel, well-executed signal for detecting compositional reasoning failures, backed by comprehensive experiments and rigorous statistics.

Takeaways

• Decompose and check: For any multi-step reasoning task, you can derive a cheap consistency check by asking the model to decompose the problem and comparing the direct answer with the composed answer. This can be added on top of existing confidence pipelines with no training.

• Use own CoT as decomposition: The paper shows that extracting sub-questions from the model’s own chain-of-thought works well (even when the CoT is imperfect). Practitioners can reuse existing CoT prompts.

• Selective prediction lift: The OC signal improves accuracy-at-fixed-coverage over tuned CoT-SC when both use the same K=3 budget. This means you can filter low-OC samples and achieve higher accuracy on the remaining queries—a practical tool for deployment.

• Transferable framing: The idea of “structural consistency” is broader than QA. It likely applies to instruction following, tool use, and any pipeline where a model’s output should be invariant under decomposition and recomposition.