From Plans to Pixels: Learning to Plan and Orchestrate for Open-Ended Image Editing

Paper: 2605.15181 Authors: Anirudh Sundara Rajan, Krishna Kumar Singh, Yong Jae Lee Categories: cs.CV

The Gap

Modern image editing models like Stable Diffusion or DALL-E handle concrete instructions well (“make the sky blue”), but collapse on abstract, multi-step requests (“make this advertisement more vegetarian-friendly”). Prior agent-based approaches (LLM-based decomposition, tool-use pipelines) either hardcode the decomposition logic or imitate teacher demonstrations, creating two problems: (1) planning is decoupled from actual editing outcomes—a plan that sounds good might produce terrible results, and (2) the system can’t learn from its mistakes because success is defined by matching teacher behavior, not by whether the edit actually worked.

Problem: Abstract instructions need multi-step reasoning
   |
   v
Assumption: Decomposition + tool selection can be learned from outcomes
   |
   v
Method: Planner generates atomic steps -> Orchestrator executes with tools
        -> Judge scores results -> Rewards refine both components
   |
   v
Evidence: Outperforms single-step and rule-based baselines on coherence
   |
   v
Conclusion: Coupling planning with reward-driven execution improves reliability

The Increment

One sentence: Before this paper, multi-step image editing relied on handcrafted pipelines or teacher imitation; after, both planning and execution can be learned end-to-end from outcome-based rewards.

Core Mechanism

The system has three components. A planner (fine-tuned LLM) takes an abstract instruction and generates a sequence of atomic editing steps in natural language. An orchestrator (trained policy network) reads each step, selects which editing tool to use (inpainting, style transfer, object removal, etc.), and determines which image region to apply it to. A vision-language judge (VLM-based reward model) evaluates the final result against the original instruction, scoring both instruction adherence and visual quality.

The orchestrator is trained via reinforcement learning to maximize judge rewards. Successful editing trajectories (high-reward sequences of tool selections and region masks) are collected and used to fine-tune the planner, teaching it to generate decompositions that lead to executable, high-quality edits. This creates a feedback loop: better plans enable better execution, and better execution data improves planning.

Instruction: "Make ad vegetarian-friendly"
   |
   v
[Planner (LLM)]
   |
   +---> Step 1: "Remove meat from plate"
   +---> Step 2: "Add vegetables to plate"
   +---> Step 3: "Change text to 'Plant-Based'"
   |
   v
[Orchestrator (Policy)]
   |
   +---> Step 1: Tool=Inpaint, Region=<plate area>
   +---> Step 2: Tool=ObjectAdd, Region=<plate area>
   +---> Step 3: Tool=TextEdit, Region=<text box>
   |
   v
[Judge (VLM)] --> Reward: 0.87
   |
   v
Update Orchestrator (RL) + Refine Planner (successful trajectory)

Think of this like a kitchen brigade learning a new recipe. The head chef (planner) writes down the cooking steps. The line cooks (orchestrator) execute each step, choosing which tools (knife, pan, oven) and ingredients (regions) to use. The food critic (judge) tastes the final dish and gives a score. The line cooks adjust their technique based on the score (RL training). When a dish scores well, the head chef studies what steps led to success and updates the recipe book (planner refinement). Over time, the recipe becomes more executable, and the cooks become better at following it. The key insight: the recipe and the execution technique co-evolve based on whether the dish actually tastes good, not whether it matches some textbook procedure.

Key Concepts

• Atomic decomposition: Breaking a complex instruction into the smallest meaningful editing operations. “Make this vegetarian-friendly” becomes three atomic steps: remove meat, add vegetables, change text. Each atomic step should be simple enough that a single tool can handle it. The challenge is that “atomic” depends on what tools you have—if you only have an inpainting tool, “add vegetables” might need further decomposition into “generate vegetable image” + “composite onto plate.” This paper assumes a fixed tool library, so atomicity is defined relative to those tools.

• Outcome-based rewards: Instead of judging whether the agent followed the right process (imitation learning), judge whether the final image satisfies the instruction. The judge is a vision-language model that scores two things: (1) does the edited image match the instruction? (2) does it look realistic? This shifts the optimization target from “do what the teacher did” to “produce results that work.” The risk is that the judge itself might be miscalibrated, rewarding superficial changes over meaningful edits.

• Experiential refinement: The planner improves by studying its own successful executions, not by imitating a teacher. When the orchestrator achieves a high reward, that trajectory (instruction → decomposition → tool sequence → final image) is added to the planner’s training data. This is different from standard imitation learning because the “teacher” is the agent’s own past successes, filtered by outcome quality. It’s a form of self-play where the planner learns what kinds of decompositions actually lead to good edits.

Framework Shift

Before (rule-based or imitation):        After (this paper):

Instruction                              Instruction
    |                                        |
    v                                        v
[Handcrafted Rules]                      [Planner (learnable)]
    |                                        |
    v                                        v
Fixed Decomposition                      Atomic Steps
    |                                        |
    v                                        v
[Tool Pipeline]                          [Orchestrator (RL)]
    |                                        |
    v                                        v
Execute All Steps                        Execute + Get Reward
    |                                        |
    v                                        v
Output (no feedback)                     [Judge] --> Reward
                                             |
                                             v
                                         Refine Both Components

From fixed pipelines to adaptive co-evolution, the core shift is closing the loop between planning and execution through outcome-based learning.

Expert Assessment

Problem choice: Real gap. Abstract instructions are where current models visibly fail, and it’s a natural next frontier after single-step editing matured. The problem sits at the intersection of tool use, planning, and vision-language grounding—all active areas. Not manufactured.

Method maturity: Clever architecture but relies heavily on the judge’s quality. If the VLM judge is biased or shallow (e.g., rewards superficial changes), the whole system optimizes for the wrong thing. The paper doesn’t deeply interrogate judge failure modes. The RL component is standard policy gradient—no novel algorithmic contribution there. The experiential refinement loop is the genuinely interesting idea, but it’s underexplored: how many iterations does it take to converge? Does the planner overfit to the orchestrator’s quirks?

Experimental integrity: Baselines are reasonable (single-step models, rule-based decomposition, imitation learning). The human evaluation is small-scale but appropriate for this task. One red flag: no ablation on judge quality. What happens if you swap in a weaker VLM? The results suggest the method works, but the error bars are wide, and some qualitative examples show the system still struggles with spatial reasoning (e.g., “move object to the left” sometimes fails).

Writing quality: The abstract and intro are strong. The method section is dense—too much notation for what’s conceptually a simple feedback loop. Figure 2 (the system diagram) does most of the explanatory work; the text should lean on it more. The related work section is thorough but could be trimmed. The biggest missed opportunity: no discussion of failure modes or limitations until the appendix. A paragraph in the main text acknowledging where the judge breaks down would strengthen credibility.

Verdict: weak accept — Solid contribution to an important problem, but the reliance on judge quality is underexplored, and the experimental scale is modest.

Takeaways

Steal the feedback loop structure: The idea of training a planner by filtering its own successful executions (not imitating a teacher) transfers to any multi-step decision problem. If you’re building an agent that decomposes tasks, don’t just imitate demonstrations—run the decompositions, measure outcomes, and retrain on what worked.

Judge design matters more than you think: The quality ceiling of this entire approach is set by the judge. If you’re building a similar system, invest heavily in judge robustness. Test it adversarially: can it detect superficial edits that technically satisfy the instruction but miss the intent?

Atomic decomposition is tool-dependent: Don’t treat “atomic” as an absolute property. A step is atomic relative to your tool library. If you add more powerful tools, you can make the decomposition coarser. If your tools are weaker, you need finer steps. Design the planner and tool library together, not separately.