Recursive Multi-Agent Systems: Scaling Collaboration Through Latent-Space Loops

Paper: 2604.25917 Authors: Xiyuan Yang, Jiaru Zou, Rui Pan, Ruizhong Qiu, Pan Lu, Shizhe Diao, Jindong Jiang, Hanghang Tong, Tong Zhang, Markus J. Buehler Categories: cs.AI, cs.CL, cs.LG

The Gap

Multi-agent systems (MAS) have agents collaborate by exchanging text messages. Recursive language models scale single-model reasoning by iteratively refining latent states. But these two scaling axes—multi-agent collaboration and recursive computation—have remained separate. Existing MAS waste tokens on serializing thoughts into text, then deserializing back into latent states. Recursive models stay single-agent. No one has asked: can we make the collaboration itself recursive, operating directly in latent space?

Problem: MAS uses text exchange     Recursive models stay single-agent
         (token-heavy, slow)        (no collaboration scaling)
                    |                           |
                    +------------+-------------+
                                 |
                    Assumption: Collaboration can be
                                a latent-space loop
                                 |
                    Method: RecursiveMAS framework
                            - RecursiveLink module
                            - Inner-outer loop training
                                 |
                    Evidence: 8.3% accuracy gain
                              2.4x speedup, 75% fewer tokens
                              across 9 benchmarks
                                 |
                    Conclusion: Recursive collaboration
                                beats text-based MAS

The Increment

One sentence: Before this paper, agents collaborated by passing text messages; after, they can collaborate through recursive latent-space loops with gradient-based co-optimization.

Core Mechanism

RecursiveMAS treats the entire multi-agent system as a unified recursive computation over latent states. Instead of agents generating text to communicate, they pass hidden states directly through a RecursiveLink module. This module has two jobs: generate in-distribution latent “thoughts” for the next agent, and transfer cross-agent latent states without serialization overhead.

The system runs in rounds. Each round, agents process their inputs, update their latent states, and pass those states to the next agent via RecursiveLink. After multiple rounds, the final agent produces an output. The whole loop is differentiable, so you can backpropagate through all rounds and all agents simultaneously.

Training uses an inner-outer loop structure. The inner loop optimizes individual agents on their subtasks. The outer loop optimizes the entire system end-to-end, using shared gradient-based credit assignment to figure out which agent contributed what to the final answer. This co-optimization means agents learn to collaborate, not just perform their isolated roles.

Round 1:          Round 2:          Round 3:
Agent A           Agent A           Agent A
  |                 |                 |
  v                 v                 v
[latent]  -->  [latent]  -->  [latent]  -->  Output
  |                 |                 |
RecursiveLink   RecursiveLink   RecursiveLink
  |                 |                 |
  v                 v                 v
Agent B           Agent B           Agent B
  |                 |                 |
  v                 v                 v
[latent]  -->  [latent]  -->  [latent]

Gradients flow backward through all rounds and agents

Think of it like a relay race where runners pass a baton, but instead of physically handing it off, they telepathically transfer their momentum and strategy. Each runner (agent) doesn’t just run their leg—they adjust their pace and technique based on what the previous runner transferred and what the next runner needs. The coach (training algorithm) watches the whole race, figures out which runner’s adjustment helped or hurt the final time, and trains everyone accordingly. The baton (latent state) carries not just position but accumulated understanding, and it never gets dropped because there’s no physical handoff—just a continuous flow of internal state.

Key Concepts

• Latent-space collaboration: Traditional MAS has agents write out their thoughts as text, send that text to the next agent, who reads it and converts it back into internal representations. This is like two people who speak the same language being forced to communicate by writing letters in a foreign language, then translating back. Latent-space collaboration skips the translation: agents pass their internal representations directly. Imagine Agent A has a 768-dimensional vector representing “the user probably wants a mathematical proof.” Instead of generating the text “I think this needs a proof,” then having Agent B tokenize and embed that text, Agent A just hands over the vector. Agent B’s RecursiveLink module takes that vector and generates a new latent thought that’s compatible with Agent B’s processing style, preserving the semantic content but adapting the format.

• Recursive computation as scaling axis: Scaling usually means bigger models or more data. Recursive computation scales by depth—running the same model multiple times, each time refining the previous output. It’s like editing a draft: first pass gets the structure, second pass fixes logic, third pass polishes language. RecursiveMAS extends this from one model to multiple agents. The system doesn’t just refine one agent’s output; it refines the collaboration itself. Each round, agents get better at understanding what their partners need and how to contribute. The recursion isn’t just “try again”—it’s “try again with accumulated cross-agent context.”

• Inner-outer loop co-optimization: Training a multi-agent system is tricky because you need to assign credit: if the final answer is wrong, which agent messed up? The inner loop trains each agent on its specific subtask (e.g., Agent A learns to retrieve relevant facts, Agent B learns to reason). The outer loop trains the whole system end-to-end, backpropagating through all agents and all rounds. This is like training a sports team: individual drills (inner loop) improve each player’s skills, but scrimmages (outer loop) teach them to work together. The key insight: because everything is differentiable, gradients flow backward through the entire collaboration graph, so each agent learns not just “do my job well” but “do my job in a way that helps my teammates.”

Framework Shift

Before (text-based MAS):              After (RecursiveMAS):

Agent A                               Agent A
  |                                     |
  v                                     v
[generate text]                       [latent state]
  |                                     |
  v                                     v
"I think we need..."  ---------->   RecursiveLink
  |                                     |
  v                                     v
[tokenize + embed]                  [latent thought]
  |                                     |
  v                                     v
Agent B                               Agent B
                                        |
Single pass, no backprop            Recursive rounds,
across agents                       end-to-end gradients

From sequential text exchange to recursive latent-space loops, the core shift is treating collaboration as a differentiable computation graph rather than a message-passing protocol.

Expert Assessment

Problem choice: Real gap. Multi-agent systems and recursive models are both active research areas, but they’ve been siloed. The observation that text-based communication is a bottleneck is valid—serialization overhead is measurable, and latent-space transfer is a natural next step. This sits at the intersection of two scaling trends, which is a smart place to be.

Method maturity: The core idea (latent-space agent communication) is elegant, but the execution leans on existing techniques (LoRA adapters, standard backprop). RecursiveLink is essentially a learned projection layer—not groundbreaking, but sufficient. The inner-outer loop training is borrowed from meta-learning. The real contribution is the integration: showing that you can make the whole thing differentiable and train it end-to-end. That said, the paper doesn’t explore failure modes deeply. What happens when agents diverge in latent space? How sensitive is this to initialization?

Experimental integrity: Baselines are reasonable (GPT-4, Claude, existing MAS frameworks, recursive single-agent models). The 8.3% average improvement is consistent across 9 benchmarks, which is reassuring. The speedup (2.4×) and token reduction (75%) are architectural wins, not cherry-picked. However, the paper doesn’t report variance or statistical significance tests. The ablation studies are present but shallow—would’ve liked to see more on what happens when you remove the outer loop or use random projections instead of learned RecursiveLink.

Writing quality: The paper front-loads motivation well, but the method section is dense. The recursive computation formalism (Section 3.2) could be clearer—too many subscripts, not enough intuition. The related work section is thorough but reads like a checklist. The biggest missed opportunity: no discussion of when this approach fails or what types of tasks benefit most from recursive collaboration. The conclusion is generic.

Verdict: weak accept — Solid contribution at the intersection of two active areas, with consistent empirical gains, but lacks depth in failure analysis and theoretical grounding for when/why recursion helps.

Takeaways

Latent-space communication is underexplored: If you’re building any system with multiple models or agents, consider whether they really need to exchange text. Passing hidden states directly can save tokens and preserve semantic richness. This applies beyond MAS—think multi-stage pipelines, ensemble methods, or even human-AI collaboration where the AI maintains internal state across turns.

Differentiable collaboration graphs: The idea of treating an entire multi-agent system as a single differentiable computation graph is portable. If you’re training agents to work together, don’t just optimize them individually—backprop through the whole collaboration. This requires careful design (everything must be differentiable), but the payoff is agents that learn to complement each other rather than just coexist.

Recursion as a scaling knob: When you hit a performance ceiling, consider depth (recursive refinement) before width (more agents or bigger models). This paper shows that iterating over the same collaboration structure can yield gains comparable to adding more agents. The trick: make sure each round has access to accumulated context, not just the previous round’s output.

Inner-outer loop training: If you’re training a modular system (agents, pipeline stages, ensemble components), separate task-specific training (inner loop) from system-level training (outer loop). This prevents the system from collapsing into a single monolithic model while still enabling end-to-end optimization.