J-Comm: Jacobian-Guided Latent Communication between LLM Agents

Paper: 2610.32871
Authors: Xiangwei Wang, Wei Wang, Sachith Seneviratne, Ken Chen, Yu Xia, Peng Wang, Saman Halgamuge
Categories: cs.CL, cs.MA

The Gap

Multi-Agent Systems (MAS) powered by collaborating Large Language Models have unlocked unprecedented capabilities in complex planning, formal verification, and distributed problem solving.

However, virtually all existing systems communicate via TextMAS (Natural Language Text Messaging):

1. The Verbose Token Tax: Agents generate thousands of natural language tokens merely to negotiate, summarize, or query intermediate progress. This token tax inflates inference latency and API costs.
2. The Discretization Information Loss: Thought processes inside modern Transformers exist as continuous, high-dimensional probability distributions and latent trajectories. Forcing an agent to pass its intermediate thoughts through a discrete argmax or temperature-sampled text token string acts as a harsh lossy quantization step.

Why Prior Latent Communication Failed

Direct Latent MAS—passing hidden activation vectors directly between agents—theoretically solves the token tax. But in practice, it faces two insurmountable barriers:

• The Heterogeneity Wall: Hidden dimensions ($d_{\text{model}}$) are tightly bound to specific model architectures. A 14B model cannot directly send its 5120-dim hidden states into the 2048-dim layers of a 3B model without complex, brittle projection heads.
• Static Vectors Lack Intent: Passing raw hidden states only tells the receiver what the sender currently represents, not which directions matter or how the receiver should adjust its own reasoning trajectory.

   THE DUAL FAILURE OF CONVENTIONAL AGENT COMMUNICATION

   1. Text-Based MAS (Standard):
      Agent A [Rich Continuous Latents] ---> [Argmax Discretization] ---> "I think x=3 because..."
                                                                                  |
      Agent B [Must re-read & re-parse 1,000 text tokens from scratch] <----------+
      ===> Severe latency, massive compute waste, loss of intermediate nuances.

   2. Naive Latent MAS:
      Agent A (Qwen-8B, d=4096)  ---> [Raw Vector: 4096 floats] 
                                                    | (Mismatch!)
      Agent B (Qwen-4B, d=2560)  <------------------+ 
      ===> Architectural crash! Cannot ingest foreign hidden dimensions without retraining.

The Increment

One sentence: J-Comm establishes a training-free latent communication framework that combines continuous Geometric Agreement for Latent Reasoning (GEAR) with a dimension-agnostic Hidden-and-Jacobian (H+J) interface, enabling heterogeneous LLM agents to exchange continuous directional thoughts across differing model widths without KV-cache sharing.

Core Mechanism

J-Comm formulates latent agent collaboration through two synchronized mathematical innovations:

1. GEAR (Geometric Agreement for Latent Reasoning):
   • A training-free continuous reasoning module inside the sender agent.
   • Instead of greedily emitting discrete tokens, GEAR reweights candidate tokens over existing embedding manifolds, monotonically optimizing geometric consensus among candidate hypotheses at each intermediate step in continuous embedding space.
2. The Dual H+J Communication Interface:
   • Rather than sending raw dimension-locked activations, J-Comm constructs two complementary semantic channels:
     • The H-Channel (Continuous Hidden State): Supplies continuous input representations derived from token feedback distributions, giving the receiver the factual grounding of the sender’s hypothesis.
     • The J-Channel (Jacobian Guidance Lens): Computes the first-order sensitivity of the sender’s reasoning state projected onto the vocabulary space: $J = \frac{\partial \mathcal{L}_{\text{sender}}}{\partial \mathbf{h}_{\text{sender}}} \in \mathbb{R}^{|\mathcal{V}|}$
     • The Genius of Vocabulary Coordinates: Because the vocabulary space $\mathcal{V}$ is shared across model families (or easily aligned via standard tokenizers), the Jacobian gradient $J$ acts as a universal, model-width-independent coordinate system that conveys which semantic directions the sender found most sensitive!
3. Universal Reconstruction and Prefill Injection:
   • The Receiver agent receives the H and J signals and reconstructs them using its own native embedding dictionary.
   • It directly injects these directional cues into its own intermediate transformer layers during the prompt prefill phase.
   • Requires zero KV-cache sharing and introduces zero additional training parameters.
4. Empirical Benchmarks:
   • Tested on heterogeneous agent pairs (e.g., Qwen3-8B communicating directly to Qwen3-4B) across seven standard benchmarks:
     • Achieves 94.80% on MATH-500, 54.55% on GPQA-Diamond, 73.33% on AIME 2024/2025, and 62.93% on ALFWorld.
     • Slashes generated communication tokens by 19.2% to 74.8% while outperforming TextMAS by +1.29 to +8.59 percentage points!

   J-COMM H+J LATENT COMMUNICATION ARCHITECTURE

   [Sender Agent: Large Model, e.g. Qwen3-8B]
   - Continuous reasoning via GEAR manifold consensus
   - Extracts:
     1. H-Channel: Continuous hidden state distributions
     2. J-Channel: Vocabulary-indexed Jacobian gradient lens: d(Loss) / d(h)
                         |
                         | [Vocabulary-Indexed Coordinate Space: No dim mismatch!]
                         v
   [Receiver Agent: Smaller Model, e.g. Qwen3-4B]
   - Reconstructs H and J cues through native embedding dictionary
   - Injects directional guidance directly into intermediate layers during Prefill
                         |
                         v
   [Ultra-Fast, High-Precision Downstream Output]
   - Up to 74.8% fewer tokens generated
   - Direct mathematical consensus without verbose text chats!

Key Concepts

• Geometric Agreement for Latent Reasoning (GEAR): An optimization protocol that seeks continuous geometric alignment among multiple candidate hypotheses within token embedding space.
• Jacobian Lens: The projection of intermediate hidden-state gradient sensitivity onto vocabulary coordinates, representing functional reasoning momentum independently of model width.
• Prefill Injection: The technique of modulating intermediate transformer layer activations during the initial prompt encoding phase rather than through sequential autoregressive generation.

Framework Shift

Before (The Dichotomy of Multi-Agent Communication):
  TextMAS: Human-like text chats -> Verbose token taxes, lossy discrete quantization.
  Prior Latent MAS: Raw activation sharing -> Fails across different model sizes, zero intent.

After (J-Comm Jacobian Latent Synergy):
  Communication Form: Continuous H (Content) + J (Directional Intent in Vocab Space).
  Heterogeneity Freedom: Qwen-8B talks smoothly to Qwen-4B without retraining or KV sharing.
  Efficiency & Power: Slashes token usage by up to 75% while beating text-based accuracy.

Expert Review

Topic Value: Visionary. Transitioning multi-agent collaboration from slow, verbose text chats to high-bandwidth, continuous latent communication is the inevitable future of AI systems.

Method Maturity: Grounding the Jacobian sensitivity matrix in vocabulary space to solve the model-width heterogeneity problem is a masterstroke of mathematical ingenuity.

Experimental Rigor: Rigorously validated on the most challenging frontier reasoning benchmarks (AIME 2024/2025, MATH-500, GPQA-Diamond), establishing consistent gains over text baselines across heterogeneous agent architectures.

Writing Quality: Beautiful mathematical structure, clear geometric intuitions, and extensive ablations demonstrating the distinct roles of the H and J channels.

Overall Verdict: strong accept — A pioneering framework that will catalyze the transition toward next-generation continuous latent multi-agent systems.

Takeaways

• LLM agents do not need to speak English or generate full text sentences to collaborate effectively; latent vectors convey richer meaning with a fraction of the compute.
• Using vocabulary-projected Jacobians solves the longstanding model-heterogeneity barrier, allowing models of different sizes to share continuous reasoning guidance.
• Combining continuous state representations (H) with directional sensitivity gradients (J) gives receiver agents both the context and the momentum needed to solve complex mathematical problems.