Empowering GUI Agents via Autonomous Experience Exploration and Hindsight Experience Utilization for Task Planning

Paper: 2606.27330
Authors: Tianyi Men, Zhuoran Jin, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao
Categories: cs.CL, cs.AI, cs.CV, cs.LG

The Gap

Existing work on multimodal web agents mostly relies on large proprietary models (GPT-4V, Gemini) or distillation from them.
Small open-source MLLMs (7B–13B) are appealing for cost and privacy, but they suffer from two crippling defects:
weak task planning (they cannot break a complex instruction into a coherent sequence of actions) and limited cross-website compositional generalization (they fail when the website layout or sequence of steps differs from training).
Prior attempts to improve planning involved either manual annotation of decomposed tasks (expensive and not scalable) or model-based synthesis that used a strong teacher to generate flat action sequences, which ignored the hierarchical structure of tasks and produced low-quality high-level plans.

The logical path this paper traverses:

Problem: small MLLMs plan poorly across websites
    |
    +-- Assumption: autonomous exploration in real GUI environments can acquire planning experience
    |       +-- but raw experience is noisy and often fails
    |       v
    v   Method: PEEU - two-phase pipeline
    +------> [Phase 1] Exploration: agent tries tasks -> records success/failure + state transitions
    |               |
    |               v
    |       [Phase 2] Utilization: hindsight relabeling of failures -> synthesize high-level task decompositions
    |               |
    |               v
    +------> Training on hindsight data improves planning without external supervision
                |
                v
    Evidence: 30.6% on real-world benchmark, beating Qwen2.5-VL-32B (which is 4x larger)
                |
                v
    Conclusion: constructing high-level hindsight tasks and leveraging experience is crucial for OOD planning

The Increment

One sentence: Before PEEU, small MLLMs could only imitate flat action sequences from static datasets and barely generalized to unseen websites; after PEEU, they autonomously discover and learn hierarchical planning strategies from their own successes and failures, matching or surpassing models 4x their size.

Core Mechanism

PEEU has two main components. The Explorer first deploys the small MLLM in a real GUI environment (simulated browser) and lets it freely attempt a set of seed tasks. For each attempt, the Explorer records the initial instruction, the sequence of actions taken, the states visited, and the outcome (success or failure). All traces are dumped into an Experience Buffer.

Next, the Hindsight Utilizer reads the buffer. For failed traces, it uses the final state (which might still contain useful information) to relabel the original goal with a *high-level task decomposition. For example, if the task was “book a flight” and the agent failed at step 3, the hindsight relabeler rewrites the trace as a correctly executed hierarchical plan: “step 1: open booking page, step 2: fill departure, step 3: fill destination (original failure point relabeled as correct)”, treating the entire sequence as a success for a different but valid high-level task. This produces training data that is clean, hierarchical, and aligned with the agent’s own capabilities.

The final step is training the original small MLLM on these hindsight-generated data using standard instruction fine-tuning.

[Phase 1: Exploration]                    [Phase 2: Hindsight Utilization]
+------------------+                     +------------------+
| GUI Environment  |                     | Experience Buffer|
| (simulated page) | <-- actions         | (traces: success |
+--------+---------+     from agent      |  + failure)       |
         |                               +--------+---------+
         v                                        |
+------------------+                     +--------v---------+
| Small MLLM agent |                     | Hindsight        |
| (attempts tasks) |                     | Relabeler        |
+--------+---------+                     | (rewrites failed |
         |                               |  traces as high- |
         +--> state, action, result --->  |  level successes)|
         |                               +--------+---------+
         v                                        |
+------------------+                     +--------v---------+
| Experience Record|                     | Training Data    |
| (buffer)         |                     | (hierarchical    |
+------------------+                     |  decompositions) |
                                          +--------+---------+
                                                   |
                                                   v
                                          +------------------+
                                          | Small MLLM       |
                                          | (instruction     |
                                          |  fine-tuned)     |
                                          +------------------+

Think of PEEU as a personal chef who learns to cook multi-course meals by experimenting in their own kitchen.

• The GUI Environment is the kitchen with tools (web elements).
• The Explorer is the chef trying random recipes (tasks) — sometimes they burn the steak (failure), sometimes they plate perfectly (success).
• The Experience Buffer is the chef’s notebook where every mishap and triumph is recorded.
• The Hindsight Utilizer is an experienced mentor sitting next to the chef. When the chef fails, the mentor says: “What if you had arranged the cooking order differently? Let’s rewrite this evening’s menu so that your attempted actions become the *correct plan for a different but equally impressive dish.”
• The final training is the chef practicing the rewritten menus until those high-level sequences become natural.

The key insight: the chef never needs a textbook (manual decomposition) or a demonstration from a Michelin-star chef (large model). They learn from their own kitchen messes, abstract them into proper plans, and get better at planning whole meals rather than just chopping vegetables (low-level actions).

Key Concepts

• Hindsight Experience Relabeling: A technique borrowed from robotics (Andrychowicz et al., 2017). For a failed attempt, you change the *goal post-hoc to match what the agent actually achieved, then treat the trajectory as a success for that new goal. In this paper, instead of relabeling low-level goals, they relabel the task decomposition — turning a failed sub-step into a valid high-level step. Example: agent tries to click “Checkout” but fails because page hasn’t loaded. Hindsight relabel says: “the correct high-level step was ‘wait for page load’, and the click attempt is actually a correct execution of that step.” This creates positive examples from negative experiences.

• Compositional Generalization at Multiple Granularities (TDHAF): The paper proposes a framework to measure how well a model’s planning ability transfers across three levels: low (atomic skills like clicking, typing), medium (single-page sequences), high (multi-page workflows). They discover that mastering low-level skills does *not guarantee high-level planning — a finding that challenges the common practice of only training on atomic actions. Their TDHAF analysis shows that training on high-level hindsight tasks directly improves out-of-distribution (OOD) planning because the tasks force the model to learn structured reasoning rather than rote execution.

• Autonomous Exploration: The agent is dropped into a website without any predefined curriculum or human intervention. It chooses which tasks to attempt (from a seed list), explores the page freely, and may fail many times. This is essential because it generates realistic failure modes that no static dataset can cover. The failures are then the raw material for hindsight learning.

Framework Shift

Before PEEU, the dominant paradigm for small MLLM planning was supervised fine-tuning on static, flat action sequences extracted from large models or manual annotation. After PEEU, the paradigm becomes self-supervised exploration + hindsight data generation on hierarchical task decompositions.

Before (mainstream approach):                  After (this paper):
+------------------+                          +------------------+
| Large teacher    |                          | GUI Environment  |
| (GPT-4V) generates|      --->               | (real websites)  |
| flat action seqs |                          +--------+---------+
+--------+---------+                                   |
         |                                    +--------v---------+
         v                                    | Small MLLM       |
+------------------+                          | explores freely  |
| Small MLLM       |                          +--------+---------+
| copies sequences |                                   |
+--------+---------+                         +--------v---------+
         |                                    | Hindsight        |
         v                                    | relabeler        |
+------------------+                          +--------+---------+
| Generalization?  |                                   |
| Poor OOD         |                           +--------v---------+
+------------------+                           | Train on high-  |
                                                | level plan seqs |
                                                +--------+---------+
                                                         |
                                                         v
                                                +------------------+
                                                | Strong OOD       |
                                                | generalization   |
                                                +------------------+

One sentence: From *imitating flat actions from a static dataset to self-discovering hierarchical plans through autonomous failure and reflective relabeling, the core shift is replacing external supervision with environmentally grounded self-improvement.

Expert Assessment

Problem choice: Real gap. The field is obsessed with scaling up models for GUI agents, but cost and privacy concerns make small models necessary. The paper identifies a concrete bottleneck (planning generalization) and tackles it with a practical solution. It sits right at the intersection of embodied AI, self-supervised learning, and language model alignment.

Method maturity: Clever insight over brute force. The idea of using hindsight experience for *task decomposition (not just low-level goal relabeling) is elegant and well-adapted to the planning domain. They avoid complex architectural changes; instead, they modify the data generation pipeline. One could argue that a simpler approach would be to collect more diverse static data or use data augmentation, but those lack the failure-driven exploration component. PEEU is a well-crafted combination of existing ideas, not a radical break.

Experimental integrity: The baselines are reasonable: they compare against GPT-4V, Qwen2.5-VL (7B and 32B), and other open-source models. The result that their 7B model beats Qwen2.5-VL-32B is impressive and holds up given the attention paid to controlling for training data distribution. One red flag: they only report accuracy on one benchmark (unclear which benchmark — likely MiniWoB++ or a modified version). More diverse evaluation (e.g., real-world websites, unseen domains) would strengthen the claim. Also, they do not compare to methods that use RL or exploration reward shaping, which could be seen as missing a related approach.

Writing quality: The paper is dense but well-structured. The introduction clearly states the gap. The method section is clear but could benefit from a pseudocode or algorithm box. Where they cut corners: the TDHAF analysis framework is introduced late and feels tacked on. A separate section with detailed ablation and visualizations would elevate the paper significantly. The abstract claims “30.6% accuracy” but doesn’t specify the benchmark until later — a minor transparency issue.

Verdict: Weak accept — The contribution is solid and practically useful, but the novelty is more in application than in theory. The paper would be stronger with broader evaluation and a tighter integration of the TDHAF analysis.

Takeaways

• Practical technique: If you are building a GUI agent with a small MLLM, adopt the two-phase pipeline: first let your agent flail around in a mock environment, recording everything, then use hindsight to convert failures into high-quality planning examples. This is cheap and works.
• Transferable concept: Hindsight relabeling for hierarchical task decomposition is not limited to web agents. It can be applied to any sequential decision-making domain where you want the agent to learn planning from its own mistakes (robotics, game-playing, even code generation).
• Measurement takeaway: The TDHAF framework (low/medium/high granularities) is a useful tool for diagnosing where your model’s planning breaks. Practitioners should adopt this multi-level evaluation rather than just reporting end-to-end success rates.
• Caveat: The approach requires a safe simulation environment for exploration. If you deploy it on live websites, the agent may cause accidental actions (booking flights, deleting data). Always sandbox first.