The Physics of Multi-Turn Planning: A Controlled Lab for Studying How LLMs Learn to Plan

Paper: 2607.24720 Authors: Tianyi Men, Zhuoran Jin, Kang Liu, Jun Zhao Categories: cs.CL, cs.AI, cs.LG

The Gap

Existing research on LLM agents mostly trains on uncontrollable Internet data — web trajectories, human demonstrations scraped from the wild. The problem: we don’t actually know how planning ability gets acquired during pre-training, what post-training methods like GRPO or distillation actually do to planning, or how multi-source training interacts. Prior work either studies these stages in isolation or uses opaque environments where you can’t disentangle variables. Think of it like trying to understand gravity by watching random objects fall in a crowded room — too many confounds.

This paper builds a controlled lab (a deterministic multi-turn environment with tunable complexity) and runs a full physics-style investigation: acquire → shape → integrate, across three clean experimental stages.

Problem: LLM planning is a black box
    |
    v
Assumption: We can decompose it into 3 stages
    |          and study each in a controlled env
    v
Method: Build deterministic multi-turn environment
    |     + isolate pre-training / GRPO-OPD / MOPD
    v
Evidence:
    |  CoT state modeling > atomic skills
    |  OPD > GRPO on long-horizon low-quality
    |  MOPD: pattern compatibility determines
    |         generalization vs interference
    v
Conclusion: Planning follows "physics-like" laws
            you can discover and exploit

The Increment

One sentence: Before this paper, multi-turn planning in LLMs was trained and evaluated in uncontrolled settings with no mechanistic understanding; after this paper, we have a controlled experimental framework that reveals specific, testable laws governing how planning ability is acquired, shaped, and integrated.

Core Mechanism

The paper’s contribution isn’t a single model — it’s an experimental pipeline with three stages, each producing distinct findings.

Stage 1 — Pre-training acquisition. They construct a deterministic multi-turn environment (think: a gridworld with controllable states, actions, and transitions). They vary what data goes into pre-training: atomic skill trajectories (short, modular), long-horizon trajectories (multi-step, goal-directed), and CoT-annotated trajectories (explicit state-transition reasoning). They measure generalization to unseen compositions and longer horizons.

Stage 2 — Post-training shaping. They compare two RL post-training methods: GRPO (Group Relative Policy Optimization) and OPD (On-Policy Distillation from a teacher). Using mutual information analysis, they decompose what the student learns into “planning patterns” (transferable reasoning strategies) vs “task-specific knowledge” (memorized procedures). They map three regimes: where post-training is unnecessary (student already good), effective (post-training helps), and unsupported (post-training can’t fix the gap).

Stage 3 — Post-training integration (MOPD). They train students on *multiple teacher environments simultaneously via On-Policy Distillation. The key variable: how much do the teachers’ planning patterns overlap?

Stage 1: PRE-TRAINING ACQUISITION
    [Raw Data] --> [Controlled Env Pre-train]
         |                    |
    Atomic Skills       CoT State Transitions
    Long-horizon Traces     |
         |                  v
         +-----> [Measure: Compositional
                  Generalization]
                        |
                        v
Stage 2: POST-TRAINING SHAPING
    [Pre-trained Model] --> [GRPO] --+
         |                          |
         +----------------->[OPD]---+
                                      |
                                      v
                             [Mutual Info Analysis]
                             Planning Patterns vs
                             Task Knowledge
                                      |
                                      v
Stage 3: POST-TRAINING INTEGRATION
    [Teacher A] --+                   |
    [Teacher B] --+--> [MOPD] ------->|
    [Teacher C] --+         |         |
                            v         v
                    Compatible --> Generalization
                    Partial    --> Continual Learn
                    Conflicting --> Interference

Here’s a structural metaphor: learning to be a chef by apprenticing in multiple kitchens.

Stage 1 is your culinary school textbooks. If the textbook only has isolated techniques (chop onions, sear fish), you can execute individual skills but freeze when asked to prepare a three-course meal from scratch. But if the textbook includes even a small number of full multi-course meal workflows — showing how techniques chain together — your compositional generalization jumps dramatically. And if those textbook recipes have mistakes (suboptimal trajectories), you internalize bad habits that compound catastrophically when cooking longer meals.

Stage 2 is on-the-job training. GRPO is like learning by trial-and-error in a single restaurant kitchen — you get scored relative to your own attempts. OPD is like shadowing an experienced head chef — you learn by imitating their decision-making in real time. The paper finds that OPD works better when your starting skills are rough and the meals are complex, because the head chef provides consistent, coherent direction. But here’s the catch: if the head chef teaches you a cuisine completely different from your culinary school foundation, you might lose your original skills while not fully acquiring the new ones.

Stage 3 is apprenticing in multiple kitchens simultaneously. If all three kitchens share a common flavor-pairing logic (compatible patterns), you become a versatile chef who can generalize across cuisines. If they partially overlap, you build a growing repertoire (continual learning). But if Kitchen A says “always salt early” and Kitchen B says “never salt until the end” (conflicting patterns), you develop culinary schizophrenia and perform worse than if you’d stayed in one kitchen.

Key Concepts

• CoT State Transition Modeling: Instead of just training on input-output pairs, you train the model to explicitly narrate intermediate states: “I’m at position X, my goal is Y, so I should take action Z which moves me to state W.” This is like giving someone a GPS with turn-by-turn directions instead of just showing them the destination. The paper shows this dramatically improves long-horizon generalization because the model learns *where it is at each step, not just what to do.

• On-Policy Distillation (OPD): The teacher and student are both generating actions *in the student’s own environment (on-policy), and the student learns to match the teacher’s distribution. This is different from off-policy distillation where you just clone the teacher’s static dataset. Think of it as: OPD is watching a master chef cook in your kitchen with your ingredients, while off-policy is studying a master chef’s cookbook written for a different kitchen. OPD adapts to the student’s actual state distribution.

• Pattern Compatibility in MOPD: When you distill from multiple teachers, each teacher encodes a “planning pattern” — a latent strategy for decomposing goals. The paper uses mutual information to measure how much these patterns overlap. Compatible patterns (high overlap) = additive learning. Conflicting patterns (low overlap) = the student’s internal representation gets torn apart, like trying to simultaneously learn two incompatible game strategies.

Framework Shift

Before (mainstream approach):        After (this paper):

  Internet Data                      Controlled Environment
       |                                   |
  Train end-to-end                  Decompose into 3 stages
       |                                   |
  Evaluate on benchmarks      Pre-train --> Shape --> Integrate
       |                                   |
  "Model X scores 85%"        "Here's WHY planning works"
                              "Here's WHAT fails and when"
                              "Here's HOW teachers interact"

From “black-box training on messy data and evaluating on benchmarks” to “controlled experiments that reveal mechanistic laws of planning acquisition,” the core shift is treating LLM planning like a physics problem where you can isolate variables.

Expert Assessment

Problem choice: This is a genuine and important gap. The field has been training agents on web-scale data and hoping for the best. Nobody has seriously asked “what exactly does pre-training teach about planning, and how does post-training modify it?” The question is overdue. It sits at the intersection of mechanistic interpretability and agent training — a sweet spot.

Method maturity: Clever rather than brute force. Building a controlled environment is the kind of unsexy infrastructure work that enables real science. The mutual information decomposition for distinguishing planning patterns from task knowledge is elegant. However, the controlled environment is necessarily simpler than real-world agent tasks — there’s a risk the “physics laws” discovered here don’t transfer to open-ended settings. The authors partially acknowledge this but could be more explicit.

Experimental integrity: The baselines are reasonable — they compare GRPO vs OPD fairly under matched conditions. The three-regime characterization (unnecessary/effective/unsupported) is cleanly defined. One concern: the multi-teacher experiments only test a handful of teacher combinations. The taxonomy of compatible/partially-shared/conflicting patterns is suggestive but not rigorously defined — it’s more of a qualitative observation than a formal framework.

Writing quality: The paper is dense but well-structured. The three-stage decomposition gives it a clean narrative arc. However, the related work section is thin — they don’t adequately connect to the broader literature on curriculum learning, multi-task learning, or knowledge distillation theory. The MOPD section (Stage 3) feels rushed compared to the thoroughness of Stages 1 and 2. Rewriting Section 5 with the same rigor as Sections 3-4 would elevate the whole paper.

Verdict: weak accept — The controlled-environment methodology is genuinely valuable and the Stage 1-2 findings are solid, but the Stage 3 analysis needs more rigor and the generalization to realistic settings remains unproven.

Takeaways

1. Include even a small fraction of long-horizon trajectories in pre-training. The paper shows that atomic skills alone don’t compose — you need at least some examples showing how skills chain together. This is directly actionable for anyone curating pre-training data.

2. Use OPD over GRPO when your starting model is weak and your tasks are complex. The consistent gradient signal from teacher distillation outperforms relative reward comparison in these regimes. This is a practical training recipe, not just a research finding.

3. Audit teacher compatibility before multi-source distillation. If you’re distilling from multiple specialized models, check whether their implicit planning strategies conflict. Conflicting teachers can make the student worse than any single teacher — a critical failure mode for the “ensemble of experts” paradigm.

4. Suboptimal data is not neutral — it’s actively destructive in long-horizon settings. Errors compound over turns. Filtering for quality matters more in multi-step tasks than in single-turn settings, by a large margin.