DIRECT: When and Where Should You Allocate Test-Time Compute in Embodied Planners?

Paper: 2606.12402 Authors: Jadelynn Dao, Milan Ganai, Yasmina Abukhadra, Ajay Sridhar, Mozhgan Nasr Azadani, Katie Luo, Clark Barrett, Jiajun Wu, Chelsea Finn, Marco Pavone Categories: cs.RO, cs.AI, cs.CV

The Gap

Existing work treats test-time compute as a uniform lever: chain-of-thought depth, model size, and memory history are scaled blindly, assuming more compute = better performance. VLABench and RoboMME benchmarks show this assumption is false — gains plateau, latency balloons, and token costs explode. Prior routing methods (e.g., model selection based on input difficulty) ignore multimodal scene context and treat axes independently. The paper identifies that embodied planners need a context-aware router that decides which axis to scale and how much, based on the current visual and linguistic scene.

Problem: Naive scaling of test-time compute yields diminishing returns
  v
Assumption: Compute allocation should depend on scene context, not uniform scaling
  v
Method: DIRECT router - uses multimodal features (visual + linguistic) to select compute strategy
  v
Evidence: Pareto improvements on VLABench, RoboMME; real robot matches stronger model at 65% lower latency
  v
Conclusion: Test-time compute is a multi-axis resource; contextual routing is necessary for frontier performance at real-world cost.

The Increment

One sentence: Before this paper, embodied planners wasted test-time compute by scaling uniformly; after DIRECT, they can allocate compute intelligently per prompt, achieving frontier success rates at a fraction of the cost.

Core Mechanism

DIRECT is a lightweight routing module placed in front of the VLM planner. Given an input prompt (image + text), DIRECT extracts multimodal features via a frozen vision encoder (e.g., CLIP) and a text encoder. These features are fed into a small MLP that outputs a decision over three scaling axes: chain-of-thought depth (number of reasoning steps), model size (small / medium / large VLM), and memory history length (how many past observations to include). The router is trained via reinforcement learning (PPO) with a reward that balances success (binary) against a cost function (latency + token usage). The policy learns to choose a configuration that maximizes expected reward for each scene.

Input Prompt (image + text)
  |
  v
[Feature Extractors: frozen vision encoder + text encoder]
  |
  v
[MLP Router] --> policy outputs: {CoT depth, model size, history length}
  |
  v
[Executor: load selected VLM with config, run planning]
  |
  v
[Reward: success - cost] --> train router via PPO

Structural Metaphor: Think of the router as a *restaurant sommelier for compute. The customer (prompt) arrives with a dish (scene). A sommelier doesn’t pour the most expensive wine for everyone; they assess the dish’s spices, richness, and the guest’s preference, then recommend a wine that pairs well. The router does the same: the “wine cellar” is a set of VLM configurations (varying CoT depth, model size, history). The “tasting” is the multimodal feature extraction, and the “recommendation” is the configuration. After the meal (planning), the guest’s satisfaction (success) and the bill (cost) update the sommelier’s training. Just as a good sommelier learns which wine works for which dish, the router learns which compute axis fits which scene.

Key Concepts

• Pareto Frontier: A set of points where no configuration can improve both success and cost simultaneously. DIRECT aims to push this frontier outward: for any given cost, achieve higher success than fixed strategies. Prior work stayed inside the frontier; DIRECT breaks through by contextual allocation.
• Test-Time Compute Axes: Three levers the paper studies: (1) chain-of-thought depth (number of reasoning steps), (2) model size (parameter count), (3) memory history (how many past frames/commands to condition on). Each axis has a different cost profile and capability profile. For example, deeper CoT helps symbolic reasoning but hurts speed; larger model helps vision but costs more. The router picks the combination.
• Contextual Routing: Instead of a fixed policy (always use large model + deep CoT), the router adapts per prompt. It uses scene features (e.g., object clutter, task complexity, linguistic ambiguity) to predict which axis will matter most. This is the key epistemic insight: not all test-time compute creates equal value.

Framework Shift

Before (mainstream approach):        After (this paper):
[Fixed configuration for all prompts]  [Context-aware router per prompt]
Example: always use large model +   |  Example: small model + deep CoT for
deep CoT + long history.            |  symbolic tasks; medium model + short
                                    |  history for visual navigation.
                                    |
Cost: high for all prompts,         |  Cost: low for easy prompts, high only when needed
success: uniform but plateau        |  Success: high overall via tailored scaling
                                    |
[ASCII of fixed: all arrows same size]  [ASCII of router: variable arrow sizes]
+-->+   +-->+                       |  +-->[large CoT]-->+
                                     |  +-->[small CoT]-->+
                                     |  +--> [medium]-->+
                                     |  (arrows vary by scene)

One sentence: From uniform compute scaling to contextual compute allocation, the core shift is treating test-time compute as a multi-axis resource that must be routed intelligently per prompt.

Expert Assessment

Problem choice: Real gap. The embodied AI community is rushing to scale compute without questioning where it’s wasted. This paper calls a needed timeout. It sits at the intersection of meta-learning and resource-constrained robotics, a growing pain point as VLMs become mainstream in planning.

Method maturity: Clever insight + practical engineering. The router is lightweight (MLP, not a transformer) so it doesn’t add overhead. The RL training with cost-aware reward is straightforward but effective. Simpler approaches (e.g., threshold-based difficulty heuristics) were not tested; the paper acknowledges this but argues RL captures more complex patterns. Could be overkill for some tasks.

Experimental integrity: Baselines are fair: they compare against fixed model selections (small, medium, large) as well as random and oracle. Real robot validation with DROID setup is a big plus. Numbers hold: the success-cost Pareto plots are convincing. One red flag: the router may overfit to the benchmarks’ scene distributions; real-world deployment may need retraining. The paper doesn’t discuss generalization to unseen tasks.

Writing quality: Clear structure, decent diagrams. The “axes” concept is well-introduced. Weakness: Section 4 (experiments) is dense and could use more intuition about why certain configurations win on certain scenes. A table of success rates broken down by task category would help. The related work feels rushed — missing comparisons to recent adaptive compute works in NLP (e.g., Mixture of Depths).

Verdict: strong accept — timely, well-executed, and provides a concrete framework that makes immediately deployable engineering sense for embodied planning.

Takeaways

• Cost-aware router training: Use RL to optimize for a multi-objective (success + cost) with a customizable cost function. This idea transfers directly to any system where you have multiple model configurations or compute budgets.
• Three-axis decomposition: The insight that CoT depth, model size, and history length are *qualitatively different compute axes is valuable. Practitioners should treat them as separate knobs, not just one “compute budget” knob.
• Real robot validation: The Franka arm experiment shows the router can match a stronger model’s success with 65% lower latency. This is a recipe for deploying frontier VLMs on edge robots with limited compute.