NovaPlan: Zero-Shot Long-Horizon Manipulation via Closed-Loop Video Language Planning

Paper: 2602.20119 Authors: Jiahui Fu, Junyu Nan, Lingfeng Sun, Hongyu Li, Jianing Qian, Jennifer L. Barry, Kris Kitani, George Konidaris Categories: cs.RO, cs.AI, cs.CV

Abstract

NovaPlan addresses a fundamental challenge in robotics: enabling robots to perform complex, long-horizon manipulation tasks without prior demonstrations or training. The framework bridges the gap between high-level semantic reasoning provided by vision-language models (VLMs) and the physical grounding required for real-world robot execution. By combining closed-loop VLM planning with video generation models and geometric action extraction, NovaPlan decomposes tasks into sub-goals, generates visual predictions of outcomes, and translates these into precise robot actions. The system demonstrates autonomous error recovery through continuous monitoring and re-planning, achieving success on complex assembly tasks and the Functional Manipulation Benchmark (FMB) in a zero-shot setting.

Key Contributions

• A hierarchical framework that unifies closed-loop VLM planning with video-based action generation for zero-shot long-horizon manipulation
• A dual kinematic prior extraction mechanism that leverages both object keypoints and human hand poses from generated videos, with intelligent switching based on execution context
• Closed-loop monitoring and autonomous re-planning capabilities that enable recovery from single-step failures without human intervention
• Demonstration of complex assembly tasks and dexterous manipulation on FMB without any prior demonstrations or task-specific training

Technical Architecture

NovaPlan operates through a two-level hierarchy that separates semantic reasoning from physical execution. At the high level, a VLM planner receives natural language task descriptions and decomposes them into a sequence of sub-goals. For each sub-goal, the system generates a video prediction showing the expected outcome of the manipulation. This video serves as both a visual specification and a source of kinematic information.

The low-level execution layer extracts actionable information from the generated videos through two parallel pathways. The first pathway identifies task-relevant object keypoints and tracks their trajectories through the video frames. The second pathway detects and tracks human hand poses, which serve as natural demonstrations of manipulation strategies. A switching mechanism dynamically selects between these two kinematic priors based on factors such as occlusion levels and depth estimation confidence.

The closed-loop nature of the system is critical to its robustness. After each sub-goal execution, the VLM planner observes the actual robot state through camera feedback and compares it against the expected outcome. If discrepancies are detected, the system autonomously re-plans, either by adjusting the current sub-goal or by generating an entirely new action sequence. This feedback loop enables the system to recover from failures that would otherwise terminate execution.

Geometric Grounding and Action Computation

A key innovation in NovaPlan is how it translates visual predictions into geometrically grounded robot actions. Rather than directly imitating pixel-level video content, the system extracts sparse geometric representations that are invariant to viewpoint changes and robust to visual noise.

For object-centric manipulation, the system identifies keypoints on target objects using foundation models and tracks their 3D trajectories across video frames. These trajectories are then transformed into end-effector waypoints through inverse kinematics, with the robot following the implied motion path. This approach works well for tasks where the object geometry is clearly visible and the manipulation primarily involves moving objects through space.

For contact-rich or dexterous manipulation, human hand poses provide more informative priors. The system detects hand keypoints in the generated video and maps them to robot end-effector poses through a learned correspondence. This is particularly effective for tasks requiring specific grasp configurations or fine manipulation, where the hand pose encodes important information about contact points and force application.

The switching mechanism between these two modes is governed by confidence metrics. When object keypoints are reliably detected and tracked with low reprojection error, the system uses object-centric control. When occlusion is heavy or depth estimates are uncertain, it falls back to hand pose guidance. This adaptive strategy maintains execution stability across diverse manipulation scenarios.

Experimental Results and Analysis

NovaPlan was evaluated on three categories of long-horizon tasks: multi-step assembly, functional manipulation from the FMB benchmark, and tasks requiring error recovery. The results demonstrate significant advantages over baseline approaches that lack either closed-loop planning or geometric grounding.

On assembly tasks involving multiple objects and sequential dependencies, NovaPlan achieved success rates exceeding 75% in zero-shot settings. The system successfully completed tasks such as assembling furniture components, stacking blocks with specific ordering constraints, and constructing simple mechanical structures. Notably, these tasks required 5-8 sequential manipulation steps, with each step depending on the successful completion of previous ones.

The FMB evaluation tested the system’s ability to perform functional manipulations where the goal is defined by achieving a specific functional state rather than a geometric configuration. Tasks included opening containers, operating mechanisms, and arranging objects for specific purposes. NovaPlan’s closed-loop planning proved essential here, as the system could adapt its strategy based on observed functional outcomes rather than rigidly following a predetermined plan.

Error recovery experiments revealed the system’s robustness to execution failures. When individual manipulation attempts failed due to grasp errors, object slippage, or environmental perturbations, the VLM planner detected the discrepancy and generated corrective actions. In 68% of cases where a single step failed, the system successfully recovered and completed the overall task. This is a substantial improvement over open-loop approaches, which typically fail completely after any single-step error.

Implications for Robot Learning

NovaPlan represents a significant step toward robots that can perform complex manipulation tasks without extensive training data or task-specific programming. The zero-shot capability is particularly valuable for applications where collecting demonstrations is expensive or impractical, such as household robotics, disaster response, or space exploration.

The framework’s reliance on foundation models (VLMs and video generation models) means it inherits both their capabilities and limitations. The system can generalize to novel objects and scenarios that appear in the training data of these foundation models, but may struggle with truly out-of-distribution situations. The geometric grounding layer partially mitigates this by ensuring physical plausibility, but the high-level planning still depends on the VLM’s semantic understanding.

The closed-loop architecture suggests a path toward more autonomous robotic systems. Rather than requiring perfect execution of predetermined plans, robots can monitor their own performance and adapt in real-time. This is closer to how humans approach complex tasks—through iterative refinement rather than flawless execution. Future work could extend this approach to incorporate learning from failures, gradually improving performance through experience.

The dual kinematic prior mechanism highlights an important principle: different manipulation strategies are appropriate for different contexts. Object-centric control works well for gross motion and object transport, while hand pose guidance is better for contact-rich interaction. A mature robotic system needs multiple strategies and the intelligence to switch between them appropriately.

Takeaways

1. NovaPlan demonstrates that zero-shot long-horizon manipulation is achievable by combining VLM planning, video generation, and geometric grounding in a closed-loop framework
2. Closed-loop monitoring and autonomous re-planning are essential for robust execution, enabling recovery from single-step failures without human intervention
3. Dual kinematic priors (object keypoints and hand poses) with intelligent switching provide robustness across diverse manipulation scenarios
4. The framework achieves 75%+ success rates on complex assembly tasks and performs well on functional manipulation benchmarks without any task-specific training
5. Foundation models can provide semantic reasoning and visual prediction capabilities, but geometric grounding is necessary to translate these into physically executable robot actions