Learning from Trials and Errors: Reflective Test-Time Planning for Embodied LLMs

Paper: 2602.21198 Authors: Yining Hong, Huang Huang, Manling Li, Li Fei-Fei, Jiajun Wu, Yejin Choi Categories: cs.LG, cs.AI, cs.CL, cs.CV, cs.RO

Abstract

Current embodied LLMs excel at high-level reasoning but lack the ability to reflect on failures, causing robots to repeat mistakes rather than learn from experience. This paper introduces Reflective Test-Time Planning (RTTP), a framework inspired by human reflective practice that enables agents to learn continuously during deployment. The approach combines reflection-in-action (generating and evaluating multiple candidate actions before execution using test-time scaling) with reflection-on-action (updating both reflection models and action policies after execution through test-time training). A retrospective reflection mechanism allows agents to reassess past decisions with hindsight for proper credit assignment in long-horizon tasks. Evaluated on the newly-designed Long-Horizon Household benchmark and MuJoCo Cupboard Fitting benchmark, RTTP demonstrates substantial improvements over baseline models, with ablation studies confirming the synergistic value of both reflection modes.

Key Contributions

• Introduction of Reflective Test-Time Planning (RTTP), a dual-mode reflection framework that enables embodied agents to learn from mistakes during deployment
• Reflection-in-action mechanism using test-time scaling to generate and score multiple action candidates with internal reflections before execution
• Reflection-on-action mechanism employing test-time training to update both reflection models and action policies based on external feedback
• Retrospective reflection capability allowing agents to re-evaluate earlier decisions with hindsight for long-horizon credit assignment
• Two new benchmarks: Long-Horizon Household tasks and MuJoCo Cupboard Fitting, designed to evaluate reflective learning in embodied AI
• Comprehensive experimental validation including real-robot trials demonstrating behavioral correction through reflection

Methodology: Dual-Mode Reflection Architecture

The RTTP framework operationalizes two complementary reflection modes inspired by Donald Schön’s theory of reflective practice.

Reflection-in-Action occurs before execution and leverages test-time compute scaling. The agent generates $N$ candidate actions for the current state, then uses an internal reflection model $R_{\text{internal}}$ to evaluate each candidate by simulating potential outcomes and identifying failure modes. This produces a scored set of actions: $\{(a_1, s_1), (a_2, s_2), ..., (a_N, s_N)\}$ where $s_i$ represents the reflection score. The highest-scoring action is selected for execution, effectively implementing a best-of-N sampling strategy guided by learned reflections rather than random sampling.

Reflection-on-Action occurs after execution and employs test-time training. Upon completing an action (or subtask), the agent receives external feedback—either from environment rewards, human corrections, or automated verification. This feedback is used to compute a loss signal that updates both the reflection model $R_{\text{internal}}$ and the action policy $\pi_{\theta}$ through gradient descent. The reflection model learns to better predict which actions will succeed or fail, while the policy learns to generate better initial action candidates.

Retrospective Reflection extends reflection-on-action by allowing the agent to revisit earlier decisions in a trajectory. When a failure occurs, the agent traces back through its decision history to identify which earlier choices contributed to the failure, enabling proper credit assignment in long-horizon tasks where consequences of actions may not be immediately apparent.

Experimental Results and Analysis

The authors evaluate RTTP on two challenging benchmarks requiring long-horizon planning and error recovery.

Long-Horizon Household Benchmark: This new benchmark consists of complex household tasks requiring 10-20 steps, such as “prepare breakfast” (requiring retrieving items from multiple locations, using appliances in sequence, and handling failures like spills). RTTP achieves a 73% success rate compared to 42% for the baseline embodied LLM and 58% for a model with only reflection-in-action. The full RTTP system shows particular strength in tasks requiring error recovery—when an initial plan fails, the reflection-on-action mechanism enables the agent to adjust its strategy rather than repeating the same mistake.

MuJoCo Cupboard Fitting: This benchmark tests fine-grained manipulation and spatial reasoning by requiring agents to fit objects of varying shapes into cupboard compartments. RTTP improves success rates from 51% (baseline) to 79%, with the retrospective reflection component proving crucial for tasks where initial placement decisions affect later fitting possibilities.

Ablation Studies systematically validate each component. Removing reflection-in-action reduces performance by 18-22%, while removing reflection-on-action causes a 12-16% drop. Removing retrospective reflection particularly impacts long-horizon tasks (25% performance decrease), confirming its role in credit assignment. The studies also reveal that the two reflection modes are complementary: reflection-in-action provides immediate performance gains through better action selection, while reflection-on-action enables continuous improvement over multiple task attempts.

Real-Robot Validation: Qualitative experiments on a physical robot arm demonstrate behavioral correction. In one trial, the robot initially attempts to grasp a cup from the top (which fails due to the handle orientation). After reflection-on-action updates, subsequent attempts correctly approach from the side. The retrospective reflection mechanism proves valuable when the robot realizes mid-task that an earlier placement decision has blocked access to a required object.

Implications and Future Directions

RTTP represents a significant step toward embodied agents that learn continuously from experience rather than treating each deployment as an independent trial. The framework’s key insight—that reflection should occur both before and after action—aligns with cognitive science research on human expertise development.

The test-time training aspect is particularly noteworthy, as it enables learning without requiring offline dataset collection or simulation. This makes RTTP practical for real-world deployment where the distribution of tasks and failure modes cannot be fully anticipated during training.

However, several challenges remain. The computational cost of generating multiple action candidates and performing test-time updates may limit real-time applicability in resource-constrained settings. The reliance on external feedback signals also raises questions about how to provide appropriate supervision in unstructured environments where ground truth is ambiguous.

Future work could explore more efficient reflection mechanisms, perhaps using learned world models to simulate action outcomes rather than generating full action sequences. Integration with hierarchical planning could also improve scalability to even longer-horizon tasks. Finally, extending the framework to multi-agent settings where agents can learn from each other’s reflections presents an exciting direction.

Takeaways

1. Embodied LLMs currently lack the ability to learn from mistakes during deployment, causing repeated failures rather than accumulated experience
2. Reflective Test-Time Planning introduces dual-mode reflection: reflection-in-action (planning before execution) and reflection-on-action (learning after execution)
3. Reflection-in-action uses test-time scaling to generate and score multiple candidate actions, selecting the most promising based on internal reflections
4. Reflection-on-action employs test-time training to update both reflection models and action policies based on external feedback
5. Retrospective reflection enables agents to reassess earlier decisions with hindsight, crucial for long-horizon credit assignment
6. Experiments show 31% improvement on Long-Horizon Household tasks and 28% improvement on MuJoCo Cupboard Fitting compared to baseline embodied LLMs
7. Ablation studies confirm that reflection-in-action and reflection-on-action are complementary, each contributing unique benefits
8. Real-robot trials demonstrate concrete behavioral corrections through the reflection mechanism
9. The framework enables continuous learning during deployment without requiring offline dataset collection
10. Key remaining challenges include computational efficiency and scaling to more complex multi-agent scenarios