ATLAS: Bridging Agentic and Latent Visual Reasoning with Functional Tokens

Paper: 2605.15198 Authors: Ziyu Guo, Rain Liu, Xinyan Chen, Pheng-Ann Heng Categories: cs.CV, cs.AI, cs.CL

The Gap

Visual reasoning tasks often require intermediate visual states—think of solving a geometry problem where you need to mentally rotate shapes or overlay patterns. Current approaches split into two camps: agentic methods that call external tools (generate code, invoke APIs) to produce actual images, and latent methods that learn hidden embeddings to represent visual transformations internally.

Agentic methods are interpretable and flexible but suffer from context-switching latency—every tool call means leaving the model, executing externally, and returning. Latent methods are fast but opaque, hard to train with standard autoregressive techniques, and struggle to generalize beyond their training distribution. The field is stuck: you either pay the latency tax or sacrifice interpretability and training simplicity.

Problem: Visual reasoning needs intermediate states
   |
   +---> Agentic approach: external tool calls
   |        |
   |        +---> Interpretable but slow (context switching)
   |
   +---> Latent approach: learned embeddings
            |
            +---> Fast but opaque, hard to train/generalize
            
Gap: Can we get both speed AND interpretability?
     Can we train with standard methods?
     
ATLAS: Functional tokens = single discrete words
       |
       +---> Acts as agentic operation (interpretable)
       +---> Acts as latent reasoning unit (fast)
       +---> Standard vocabulary token (trainable via next-token prediction)
       
Evidence: Superior performance on benchmarks
          + interpretability maintained
          
Conclusion: Dual-purpose tokens bridge the divide

The Increment

One sentence: Before ATLAS, you chose between slow-but-interpretable external tools or fast-but-opaque learned embeddings; after ATLAS, a single discrete token does both jobs simultaneously.

Core Mechanism

ATLAS introduces functional tokens—ordinary vocabulary tokens that trigger internalized visual operations. When the model generates a functional token during reasoning, it simultaneously (1) signals what operation to perform (like an API call) and (2) updates the model’s internal hidden state (like a learned embedding). Crucially, these tokens require no visual supervision during training and remain standard tokens that can be predicted via next-token prediction.

The architecture is deliberately minimal. Functional tokens sit in the regular tokenizer vocabulary. During forward pass, when a functional token appears, it activates a pre-defined visual operation module (rotation, cropping, overlay, etc.) that modifies the latent representation. From the model’s perspective, it’s just generating the next token in a sequence. From the reasoning perspective, that token encodes a visual transformation.

To address training instability—functional tokens are sparse in typical sequences—the authors introduce Latent-Anchored GRPO (LA-GRPO). Standard reinforcement learning struggles when reward signals are sparse. LA-GRPO anchors functional tokens with a statically weighted auxiliary objective that provides consistent gradient updates even when functional tokens appear infrequently. Think of it as giving the model a steady compass bearing while it explores the sparse reward landscape.

Input sequence:  "Rotate the square 90 degrees clockwise"
                           |
                           v
Model generates: [text tokens] <ROTATE_CW> [text tokens]
                                    |
                    +---------------+---------------+
                    |                               |
                    v                               v
            Agentic signal:                 Latent update:
            "rotation operation"            hidden state h_t
            (interpretable)                 modified by rotation
                                            (fast, no external call)
                    |                               |
                    +---------------+---------------+
                                    |
                                    v
                            Next reasoning step
                            continues seamlessly

Think of ATLAS like a bilingual speaker switching between languages mid-sentence. When you’re bilingual, certain words in Language A might perfectly capture a concept that’s clunky in Language B, so you code-switch. The listener who understands both languages gets the full meaning instantly. ATLAS does this with “languages” of reasoning: the agentic language (explicit operations) and the latent language (internal representations). A functional token is a code-switch—it’s a word in the model’s vocabulary (so the autoregressive “listener” understands it) but it also triggers an internal operation (so the latent reasoning “listener” understands it). The model doesn’t need to translate between two separate systems; it speaks both simultaneously through the same token.

Key Concepts

• Functional Token: Imagine a keyboard shortcut. When you press Ctrl+C, two things happen: (1) the system recognizes “copy” as an operation, and (2) the clipboard state updates internally. You don’t see the clipboard change, but it’s ready for the next operation. A functional token works the same way—it’s a discrete symbol the model generates (like pressing a key), which both signals an operation name (interpretable to humans) and updates the model’s internal reasoning state (fast, no external execution). The key insight: it’s not a special architectural component; it’s just a regular token that happens to have dual semantics.

• Latent-Anchored GRPO: Standard reinforcement learning for language models struggles when the “good” tokens (functional tokens) are rare. Imagine training a dog to fetch, but you only reward it once every hundred attempts—the dog gets confused because most actions yield no feedback. LA-GRPO solves this by adding a secondary, always-present reward signal specifically for functional tokens. It’s like giving the dog a small treat every time it moves toward the ball, even if it doesn’t fetch perfectly. The “anchor” is this auxiliary objective that provides steady gradient updates, preventing the model from forgetting how to generate functional tokens during long stretches of regular text.

Framework Shift

Before (mainstream approach):          After (ATLAS):

Text reasoning:                        Text reasoning:
  "Rotate square" ----+                  "Rotate square"
                      |                        |
                      v                        v
  Generate code/API call                 Generate <ROTATE_CW>
        |                                      |
        v                                      |
  External execution                           |
  (context switch)                    +--------+--------+
        |                             |                 |
        v                             v                 v
  Return image/result          Agentic signal    Latent update
        |                      (interpretable)   (internal state)
        v                             |                 |
  Continue reasoning                  +--------+--------+
                                              |
  [Slow, interpretable]                       v
                                        Continue reasoning
  OR                                    
                                        [Fast, interpretable]
Latent reasoning:
  "Rotate square"
        |
        v
  Update hidden embedding
        |
        v
  Continue reasoning
  
  [Fast, opaque]

One sentence: From separating agentic operations (external, slow) and latent reasoning (internal, opaque) into distinct systems, to collapsing both into a single token that does both jobs at once.

Expert Assessment

Problem choice: This is a real gap. The agentic vs. latent divide has been a genuine bottleneck in visual reasoning research. The problem sits at a critical juncture—multimodal models are scaling up, and the cost of generating intermediate images is becoming prohibitive. The authors correctly identify that neither camp has a satisfying solution for production systems that need both speed and interpretability.

Method maturity: The core idea is elegant—reusing the existing token vocabulary to encode operations is clever, not brute force. However, the reliance on pre-defined visual operation modules feels like a compromise. The paper doesn’t deeply explore how these modules are designed or whether they generalize to novel operations. LA-GRPO is a solid engineering contribution, but it’s an auxiliary fix rather than a fundamental insight. A simpler approach might be curriculum learning (gradually increasing functional token density), though the authors don’t compare against this.

Experimental integrity: The baselines are fair—they compare against both agentic (code generation) and latent (learned embedding) methods. The benchmarks are appropriate (visual reasoning tasks requiring intermediate states). However, the paper lacks ablation studies on the visual operation modules themselves. Are all modules equally important? Can the model learn to compose them? The numbers look strong, but I’d want to see failure cases and out-of-distribution performance before fully trusting the generalization claims.

Writing quality: The abstract and introduction are crisp. The method section, however, rushes through the visual operation module design—this is the load-bearing component, yet it gets half a paragraph. The LA-GRPO section is clearer but could benefit from a toy example showing why standard GRPO fails. The related work section is thorough but reads like a checklist. Rewriting the method section to lead with a concrete example (e.g., “Here’s what happens when the model solves a rotation task step-by-step”) would elevate the whole paper.

Verdict: weak accept — The functional token idea is genuinely useful and the execution is competent, but the paper feels like it’s presenting a promising framework rather than a fully mature system. The lack of deep analysis on the visual modules and limited exploration of failure modes keep this from being a strong accept. Still, the core contribution is solid enough to merit publication and will likely inspire follow-up work.

Takeaways

Dual-purpose tokens as a design pattern: The idea of making a single discrete symbol serve two roles (interpretable signal + internal state update) transfers beyond visual reasoning. You could apply this to code generation (a token that both names a function and updates the program state), robotics (a token that both describes an action and updates the world model), or even dialogue systems (a token that both signals intent and updates conversation context). The key: avoid architectural bifurcation by collapsing dual semantics into the existing vocabulary.

Anchored RL for sparse signals: LA-GRPO’s trick—adding a statically weighted auxiliary objective to stabilize training when target tokens are rare—is a practical technique for any domain where you’re fine-tuning language models with RL on sparse rewards. If you’re training a model to generate specific structured outputs (SQL queries, API calls, domain-specific notation) that appear infrequently, this anchoring approach could prevent catastrophic forgetting.

Pre-defined operation modules as a pragmatic middle ground: The paper doesn’t fully commit to end-to-end learning or pure symbolic reasoning. Instead, it uses pre-defined modules as a scaffold. This is worth stealing: when building systems that need both flexibility and reliability, consider a hybrid where you hard-code the primitives but let the model learn to compose them. It’s less elegant than pure learning but more robust than pure symbolic systems.