Multi-Agent Systems Are an Information Bottleneck Problem: When Bounded Communication Helps or Hurts

Paper: 2607.16133 Authors: Wendi Yu, Lianhao Zhou, Xiangjue Dong, Sai Sudarshan Barath, Declan Staunton, Byung-Jun Yoon, Xiaoning Qian, James Caverlee, Shuiwang Ji Categories: cs.LG, cs.AI

The Gap

Multi-agent systems (MAS) built on LLMs have exploded in popularity — CrewAI, AutoGen, LangGraph, you name it. The community has run hundreds of benchmarks showing MAS sometimes beats a single-agent system (SAS), sometimes loses, and sometimes ties. But nobody had a principled answer to the question: under what conditions does splitting work across agents actually help?

Prior work falls into two camps. The engineering camp (CrewAI, MetaGPT, etc.) builds agent frameworks and reports empirical wins on specific benchmarks — but the results are brittle and don’t transfer. The theoretical camp (debate frameworks, society-of-mind models) offers qualitative intuitions about specialization and modularity, but nothing quantitative you can use to predict whether MAS will help on a new task.

This paper fills the gap by reframing the entire MAS-vs-SAS question as an information bottleneck optimization. The key structural observation: a SAS dumps its full reasoning trace into one giant context window, while a MAS splits reasoning across agents who communicate through bounded relay messages. When relay bandwidth is infinite, MAS can perfectly simulate SAS — so there’s no inherent advantage. The nontrivial case is bounded relays, where compression becomes necessary, creating a trade-off between shedding redundant context (good) and losing task-relevant information (bad).

                            The Gap and Path
                            ================

  Problem
  -------
  MAS sometimes beats SAS,
  sometimes doesn't.
  No principled "when & why"
          |
          v
  Assumption
  ----------
  SAS: one big context,
  MAS: isolated contexts
  + bounded relay messages
          |
          v
  Method
  ------
  Information bottleneck
  framework: parameter beta
  controls compression vs
  information loss trade-off
          |
          v
  Evidence
  --------
  18 controlled experiments,
  5 benchmarks, 3 model scales,
  beta predicts MAS advantage
          |
          v
  Conclusion
  ----------
  MAS helps when relay is
  near-sufficient & model is
  weak (compression > loss).
  MAS hurts when relay drops
  info & model is strong.

The Increment

One sentence: Before this paper, choosing MAS vs SAS was a vibes-based decision; after this paper, it’s an information-theoretic trade-off you can reason about and even optimize.

Core Mechanism

The paper’s framework starts with a clean structural observation. In a single-agent system, the entire reasoning chain — every thought, tool call, and intermediate result — accumulates in a single context window. The model must process all of it, even the parts that are redundant or irrelevant to the final answer. In a multi-agent system, reasoning is partitioned across agents, each with its own local context. Agents communicate only through relay messages, which are necessarily bounded in size.

The authors formalize this using rate-distortion theory and the information bottleneck. Each agent sees its local context $X$ and must compress it into a relay message $R$ of bounded capacity. Downstream agents then use $R$ to produce the task output $Y$. The information bottleneck objective minimizes $I(X; R) - \beta \cdot I(R; Y)$: minimize the bits sent through the relay while maximizing the bits relevant to the task. The parameter $\beta$ captures the effective capability of the relay channel and the model’s ability to extract useful information from compressed messages.

  SAS Architecture
  ================
  [Agent 1] ---> [Full Context: X1, X2, ..., Xn] ---> [Output]
                    (everything in one window)

  MAS Architecture
  ================
  [Agent 1] --[R1]--> [Agent 2] --[R2]--> ... --> [Output]
     |        (bounded)   |        (bounded)
     v                    v
   [Local Ctx 1]       [Local Ctx 2]

  Information Bottleneck
  ======================
  Minimize:  I(X; R)         (bits sent through relay)
  Maximize:  I(R; Y)         (task-relevant info preserved)
  Balance:   beta parameter  (capability-dependent)

Here’s the structural metaphor that makes this click. Imagine you’re managing a consulting project. In the SAS version, one consultant does everything — research, analysis, writing — and their desk is buried under every document they’ve ever touched. They could find anything they need, but wading through the pile takes time and attention. In the MAS version, you split the work across specialists who share brief memos (relay messages). If the memos are generous — say, a full dossier — the team can reconstruct anything the solo consultant knew. But if the memos are capped at one page, they must compress. Good compression (beta is high) means the right details survive. Bad compression means critical context gets lost. The paper shows that weak models are like junior consultants: the compression actually helps them focus, and they can’t effectively use the giant pile on a solo desk anyway. Strong models are like senior consultants: they’re excellent at sifting through the pile, so forcing them to work from compressed memos only hurts.

Key Concepts

• Information Bottleneck: Think of it like packing for a trip with a tiny suitcase. You can’t bring everything, so you must choose: what stuff is actually useful at your destination (task-relevant information), and what can you safely leave behind (redundancy)? The information bottleneck formalizes this as compressing your source (full context) into a compact representation (relay message) while keeping as much task-relevant signal as possible. The parameter beta is like how strict the airline is about bag size — higher beta means you care more about preserving useful stuff, lower beta means you’re more aggressive about throwing things out.

• Bounded Relay: In multi-agent systems, agents don’t share their full scratchpad — they send each other summaries. A “bounded relay” is a summary with a hard size limit. The key insight is that the *bound is what creates the trade-off. With unlimited relay, MAS is just SAS with extra steps. The bound forces compression, which is where the gains (or losses) come from. Concrete example: if Agent 1 reads 10 documents and must relay everything to Agent 2 in 500 words, those 500 words either capture the essential insight (compression helped) or miss a critical detail (compression hurt).

• Effective Parameter beta: This is the paper’s main theoretical lever. Beta captures how the balance shifts between compression benefit and information loss as a function of model capability and task complexity. When beta is high (strong models, near-sufficient relays), the model can already extract signal from noisy or redundant context, so MAS compression adds little. When beta is low (weaker models, redundant context), compression helps by reducing noise. The experimental validation across three model scales (small, medium, large) confirms this prediction: weaker models benefit more from MAS.

Framework Shift

Before (mainstream approach):       After (this paper):
                                   
  MAS vs SAS = "try both,          MAS vs SAS = information
  pick what scores higher           bottleneck optimization
  on the benchmark"                 controlled by beta
       |                                 |
       v                                 v
  [Benchmark A: MAS wins]           [Context] --compress--> [Relay]
  [Benchmark B: SAS wins]                   \
  [Benchmark C: tie]                        v
  "It depends" ----> ??            [Task Output Y]
                                   
  No theory.                       Predictive framework:
  Just vibes.                      when relay ~ sufficient
                                   & model is weak -> MAS
                                   when relay loses info
                                   & model is strong -> SAS

From “empirical vibes” to “information-theoretic optimization,” the core shift is that the MAS-vs-SAS question becomes a compression problem with a known trade-off curve.

Expert Assessment

Problem choice: This is a real and important gap. The MAS literature is drowning in framework papers that report wins on cherry-picked benchmarks without explaining *why. Any practitioner who has tried to decide between a single agent and a multi-agent setup knows this pain. The information bottleneck framing is a natural fit — perhaps too natural, in the sense that you might wonder why nobody did this earlier. It sits at the right intersection of information theory and LLM systems.

Method maturity: The theoretical contribution is clean and elegant — the simulation theorem (infinite relay = SAS simulable by MAS) is a useful null result, and the beta-parameterized trade-off is a genuine insight. However, the practical instantiation is somewhat hand-wavy. Beta is not directly measurable from a running system; it’s a post-hoc explanatory variable. The paper would be significantly stronger if they showed how to *estimate beta before deployment to make a MAS-vs-SAS decision. Right now it explains but doesn’t yet predict in the actionable sense.

Experimental integrity: 18 controlled experiments across 5 benchmarks and 3 model scales is solid for a theory paper. The experiments are designed to isolate specific variables (relay bandwidth, model capability, context redundancy) rather than just reporting aggregate accuracy. The baselines are fair — they compare SAS and MAS on identical tasks with controlled relay budgets. One concern: all benchmarks are reasoning-oriented; it’s unclear whether the framework extends to open-ended generative tasks or multi-step tool use where the “task-relevant information” is harder to define.

Writing quality: The paper is well-structured and unusually clear for a theory-heavy submission. The main theorem and its proof are accessible. However, Section 4 (experimental details) could use more space — the transition from the abstract beta framework to the concrete experimental instantiation is rushed. A dedicated section on “How to Think About Beta in Practice” would elevate the paper from theoretical contribution to actionable guidance.

Verdict: weak accept — The information bottleneck framing is the right lens and the theoretical results are clean, but the gap between the elegant theory and actionable practical guidance keeps this from being a strong accept.

Takeaways

1. Before building a multi-agent pipeline, ask: is my context actually redundant? If your single-agent prompt is already concise and relevant, splitting into agents with bounded relays will likely hurt. MAS gains come from compressing away genuine redundancy.

2. Weaker models benefit more from MAS. If you’re deploying on a smaller model, the information bottleneck compression acts as a noise filter. If you’re using a frontier model, think twice before adding agent overhead.

3. Relay bandwidth is a design knob you should think about explicitly. Don’t just default to “pass a summary.” The size and fidelity of inter-agent messages directly controls the beta trade-off. Too small = information loss. Too large = you’re just SAS with extra latency.

4. The framework gives you a language for debugging MAS failures. If your multi-agent system underperforms, check: is the relay losing task-relevant information? Is the context redundant enough to justify compression? This is more actionable than “try adding more agents.”