InSight: Making VLAs Steerable for Self-Guided Skill Acquisition

Paper: 2606.24884
Authors: Maggie Wang, Lars Osterberg, Stephen Tian, Ola Shorinwa, Jiajun Wu, Mac Schwager
Categories: cs.RO, cs.AI, cs.LG

The Gap

Existing Vision-Language-Action (VLA) models are trained end-to-end on fixed sets of demonstrations. Once trained, they can only reproduce the skills present in that data. If you want a robot to do something new — twist a bottle cap, pour liquid, sweep with a broom — you need to collect new human demonstrations, label them, and retrain. This is a dead end for open-world deployment: you cannot exhaustively demonstrate every possible skill. Prior work like RT-2 and Octo simply treats the VLA as a monolithic policy; hierarchical approaches like SayCan or Code as Policies decompose tasks but still rely on pre-programmed or prompt-engineered skill libraries. No one has tackled the problem of automatically expanding that library from the robot’s own experience.

This paper asks: can a VLA itself identify what primitive skills it is missing, attempt to acquire them autonomously, and integrate them into its own policy representation — all without human supervision of the target skill? The logical progression is:

Problem: VLA stuck with fixed skill set from training demonstrations
    |
    v
Assumption: If we decompose demonstrations into steerable primitives at a lower level,
            the model can reason about which primitives are missing for a new task
    |
    v
Method: InSight = (1) automated segmentation of demos into labeled primitives
                  + (2) VLM-guided data flywheel that detects gaps, attempts primitives,
                  + (3) automatically labels and integrates successful attempts
    |
    v
Evidence: Simulated block flipping, drawer closing, sweeping, twisting, pouring
           -- none of these target skills were shown in the original training demos
    |
    v
Conclusion: Primitive steerability enables continual skill acquisition in VLAs

The Increment

One sentence: Before InSight, VLAs could only execute skills seen in demonstrations; after InSight, they can self-discover and learn new primitives and compose them into novel long-horizon tasks with zero human demonstrations of the target skills.

Core Mechanism

InSight has two stages. Stage 1 makes the VLA steerable at the primitive level. Given a set of demonstration trajectories (end-effector poses + task descriptions), a Vision-Language Model (VLM, e.g., GPT-4V) reasons about the high-level plan and segments each demonstration into temporally ordered primitives. Each primitive gets a natural language label (e.g., “move gripper to bowl”) and a corresponding end-effector trajectory snippet. The VLA is then fine-tuned to accept a primitive label as input conditioning — essentially making each primitive a “subroutine” you can call by name. That’s the steerable part.

Stage 2 is the autonomous flywheel. When a new task instruction arrives, the robot first tries to execute it by composing known primitives. If it fails, a VLM analyzes the failure and hypothesizes which primitive is missing (e.g., “pour bottle” does not exist). The robot then autonomously attempts variations of control (using the VLM for low-level motion proposals) to perform that missing primitive. Successful attempts are automatically labeled by checking task success criteria (vision-based outcome detectors). The successful trajectory is added to the demonstration buffer, and the VLA is updated via continual fine-tuning. This loop repeats until the new task is solvable.

                          +-------------------+
                          |  Demo trajectories|
                          +--------+----------+
                                   |
                                   v
                        +----------+----------+
                        |VLM Plan Decomposer  |
                        |+ end-effector poses |
                        +----------+----------+
                                   |
                        labels + snippets per primitive
                                   |
                                   v
                        +----------+----------+
                        | Fine-tune VLA to be |
                        | primitive-conditional|
                        +----------+----------+
                                   |
              +--------------------+-------------------+
              |                                        |
              v                                        v
    +---------+--------+                   +-----------+----------+
    | Known primitives |                   | New task instruction  |
    | library          |                   +-----------+----------+
    +---------+--------+                               |
              |                                        v
              | Failed? ---> VLM identifies missing primitive
              |                    |
              |                    v
              |         +----------+----------+
              |         | Robot autonomously  |
              |         | tries control to    |
              |         | approximate missing |
              |         | primitive           |
              |         +----------+----------+
              |                    |
              |                   success? (vision check)
              |                    |           |
              |                   yes         no (iterate)
              |                    v
              |         +----------+----------+
              |         | Add demo & label    |
              |         | to data buffer      |
              |         +----------+----------+
              |                    |
              +---------+----------+
                        v
              +---------+----------+
              | Continual fine-tune|
              | VLA on updated set |
              +--------------------+

Metaphor: Think of InSight as a line cook learning a new recipe. The cook already knows basic cuts, sautéing, boiling — these are the *primitives. The head chef (VLM) hands over a recipe for a dish never cooked before. The cook tries to follow the recipe using only known moves, but at one step the recipe says “flambe” and the cook has never done that. Instead of calling the head chef, the cook’s training system (InSight) automatically detects that “flambe” is missing. The system then lets the cook experiment: pour some alcohol into a hot pan, try to ignite it, observe if flame appears (success signal). The first few tries may fail or singe the eyebrows; when a safe, successful flambe is achieved, the system records the exact motions and declares a new primitive “flambe” is now in the cook’s repertoire. Next time, the cook can flambe without thinking. The key is that the system itself decided what was missing and how to practice, without the head chef demonstrating even one flambe.

Key Concepts

• Primitive steerability: A VLA is “steerable” at the primitive level when you can invoke a specific sub-behavior (e.g., “move left”) by feeding in a text token. Without steerability, the VLA is a black box that may reproduce whole-task sequences. The paper achieves this by training the VLA to predict actions conditioned on a prefix that is the primitive label, essentially turning each primitive into a learnable subroutine with a name. Example: after training, the VLA can be prompted with “open gripper” and it executes exactly that motion regardless of context.

• VLM-guided data flywheel: A self-supervised loop where a VLM (which is a large language model with vision) reasons about task decomposition and failure modes. The VLM is not doing the fine-grained control — it’s too slow and imprecise — but it serves as a “task analyst” that decides *what to practice next. The low-level control trials are generated by the VLA itself with random or heuristic variations. The flywheel is crucial because it replaces human annotation and human demonstrations of novel skills.

• Continual fine-tuning for VLAs: Typically, fine-tuning a large VLA on new data causes catastrophic forgetting of old skills. InSight mitigates this by (1) keeping the primitive-conditioned architecture, so each primitive’s weights only need small updates, and (2) using a replay buffer of older demos. The paper shows that after learning new primitives, the model still executes old primitives correctly.

Framework Shift

Before (mainstream approach):
  Human demos of task A --> Train VLA on task A --> VLA can do task A only
  Human demos of task B --> Retrain VLA on A+B --> VLA can do A+B
  (Each new skill requires new human demos and retraining)

After (this paper):
  Generic demos (any tasks) --> InSight decomposes into primitive library --> VLA is steerable by primitive name
  New task given --> VLM detects missing primitive --> Robot self-practices primitive with auto-labeling -->
  Primitive added to library, VLA updated --> VLA can now do new task (and still do old ones)
  (No human demos of the target skill needed ever again)

One sentence: From “skill collection requires human demonstration” to “skill collection is an autonomous, closed-loop process driven by VLM analysis and self-practice,” the core shift is making primitive-level steerability the substrate for continual learning.

Expert Assessment

Problem choice: Real gap. The boundedness of VLA skills is the central obstacle to practical deployment. This paper targets a fundamental bottleneck (skill acquisition) rather than a marginal increment. It sits squarely in the “lifelong learning for robots” trajectory, which the community acknowledges as urgent but has few working systems.

Method maturity: Clever insight — the idea of using VLM to suggest missing primitives and then having the robot brute-force its own demos is not elegant but practical. The automated segmentation pipeline relies heavily on the VLM’s ability to parse demonstrations, which can be fragile (e.g., cluttered scenes, occluded grippers). The low-level control proposal for missing primitives is essentially random shooting with a VLM prior, which works for simple motions but may not scale to high-precision skills (e.g., inserting a peg). Simpler approaches like manual primitive engineering were deliberately avoided, and the paper argues they don’t scale. Fair point.

Experimental integrity: The paper claims zero human demonstrations of the five target skills (block flipping, drawer closing, sweeping, twisting, pouring). However, the *original demos used to build the primitive library likely come from some tasks — those demos are human-collected. So the “zero demo” claim is only about the target skills, not about all skills. This is honest but limits the impressiveness. Baselines are missing: they compare only against a non-steerable VLA (which obviously fails) but not against say, a purely VLM-driven robot without VLA fine-tuning. The simulated results show success rates, but we don’t see ablations on the flywheel (e.g., what if you skip the VLM analysis and just try random primitives?). The real-world experiments appear qualitative (videos), which is standard for robotics papers but weakens the quantitative claim. No red flags, but the evidence is thin on repeatability.

Writing quality: The paper is well-structured and the core contribution is clear. The weakest section is the experimental reporting: they list success rates but don’t show how many attempts were needed for the flywheel, nor do they analyze failures (e.g., what if the VLM misses a primitive?). Rewriting the experiments section with per-trial histories and convergence curves would elevate the paper significantly.

Verdict: weak accept — The idea is important and the system works in principle, but the experimental validation is too sparse and lacks strong baselines. It opens a promising direction rather than fully delivering a mature solution.

Takeaways

1. Primitive-level conditioning is a cheap way to add steerability. Practitioners can take this idea and apply it to any VLA: simply append a primitive token to the input during training and inference, then use VLM to segment demonstrations. This alone enables easy composition of skills.

2. Use a VLM as a failure analyzer, not a controller. The paper shows that a large VLM can be reliably used to diagnose *what primitive is missing from a task description, even if it cannot control the robot. This pattern (VLM as analyst, VLA as executor) generalizes: any task where you can segment actions into named blocks can benefit from this hybrid.

3. Data flywheel with automatic success detectors can replace human annotation for simple outcomes. If you have clear success criteria (e.g., drawer is closed, bottle is poured), you can auto-label attempts. This removes a major bottleneck in robotics. The key is to define success detection tightly — the paper uses vision-based checks (camera after action). This can be stolen for any repetitive skill acquisition in controlled environments.

If you have a VLA model and want it to learn new behaviors without collecting demos, InSight gives you a concrete architecture and loop. The limitations (fragile VLM segmentation, low-precision control proposal) mean you’ll need to adapt it, but the blueprint is sound.