Surgical WAM: Can Action-Free Endoscopic Video Teach a Robot to Operate?

Paper: 2608.11204 Authors: Wenrui Bao, Tianyun Jiang, Zhiben Chen, Ser-Nam Lim, Peter D. Peng, Yuzhang Shang Categories: cs.RO, cs.AI, cs.CV

The Gap

Surgical robot learning has a data problem that is different in kind, not just degree, from general manipulation. To train a policy on a dVRK (da Vinci Research Kit), you need endoscopic video that is time-synchronized with joint kinematics. That means a teleoperation setup, a trained operator, a calibrated recording pipeline, and clinical or lab access. Meanwhile, endoscopic video exists in enormous quantity — surgical archives, training libraries, public datasets — with no action labels attached whatsoever.

The field has responded to this in two mostly separate camps. One camp builds surgical world models: video generators conditioned on text or coarse controls that produce plausible endoscopic futures. SurgSora, Endora, and similar work fall here. These models are impressive as generators and get used as neural simulators or as evaluation environments — you can score a policy against predicted rollouts. But they stop at prediction. Nothing in them emits torque, joint velocity, or gripper commands, so nothing closes the loop on hardware. The other camp builds surgical imitation policies — behavior cloning, diffusion policies, ACT-style action-chunking transformers — trained directly on the scarce synchronized data. These do close the loop, but their sample efficiency is exactly as bad as the demonstration count is small, and they degrade hardest on precisely the tasks that matter clinically: contact-rich tissue interaction and bimanual coordination.

Adjacent to both, the general robotics literature has been converging on world-action models — GR-1/GR-2, UniPi, Genie-style latent-action models, and NVIDIA’s Cosmos line — where a single generative backbone predicts both future frames and actions. Cosmos Policy in particular provides the architectural template this paper builds on. But nobody had run the experiment that surgery specifically demands: hold the action-labeled budget *fixed and ask whether action-free video pretraining buys you closed-loop performance. That is a narrow, falsifiable question, and it is the one this paper answers.

  PROBLEM
  [ dVRK video+kinematics pairs are scarce and expensive ]
  [ endoscopic video alone is cheap and abundant        ]
                     |
                     v
  PRIOR WORK SPLITS TWO WAYS
  [ surgical world models ]        [ surgical imitation ]
  [ predict video only    ]        [ closes loop, but   ]
  [ used as sim / eval    ]        [ data-starved       ]
        |                                  |
        +--------------+-------------------+
                       |
                       v
  ASSUMPTION
  [ visual dynamics of surgical scenes are a          ]
  [ transferable prior: knowing how tissue, needles,  ]
  [ and tools move constrains what actions make sense ]
                       |
                       v
  METHOD
  [ Surgical WAM on Cosmos Policy backbone            ]
  [ stage 1: action-free video pretraining            ]
  [ stage 2: fine-tune on FIXED action-labeled budget ]
  [ deploy: receding-horizon closed-loop controller   ]
                       |
                       v
  EVIDENCE
  [ 4 simulated surgical tasks                        ]
  [ 63.5% -> 77.8% average success                    ]
  [ PegTransfer +20 pts absolute                      ]
  [ largest gains on contact-rich / bimanual          ]
                       |
                       v
  CONCLUSION
  [ action-free video is a usable substitute for      ]
  [ some action supervision in surgical control       ]

The Increment

One sentence: Before this paper, surgical video models predicted futures you could watch and surgical policies acted on data you could not afford; after it, there is one model that learns dynamics from unlabeled video and then converts that knowledge into executable action chunks, with a controlled experiment showing the transfer is real under a fixed label budget.

Core Mechanism

Surgical WAM is a single generative transformer with two output heads sharing one representation of the scene. Given a short history of endoscopic frames (plus a task instruction), the model predicts two things jointly: the next stretch of future observations, and an action chunk — a sequence of dVRK commands covering roughly the same time span. The joint prediction is the whole point. If the model is asked to say “the left grasper will close on the peg and the peg will lift” in pixels *and in joint coordinates simultaneously, the pixel objective forces the shared representation to encode contact, occlusion, and object motion, and the action head gets to reuse all of it.

Training runs in two stages. Stage one uses only action-free endoscopic video: no kinematics needed, so the data volume can be large. Here the model does pure future-frame prediction, learning that tissue deforms rather than translates rigidly, that instruments occlude and then reappear, that a grasped object follows the tool tip, that specular highlights move with the endoscope. Stage two switches on the action head and fine-tunes on the fixed, small budget of synchronized video-kinematics demonstrations. The action head is not learning surgical physics from scratch — it is learning a readout from an already-structured latent space into joint space. That is a much lower-dimensional learning problem, which is why a small budget suffices.

Deployment is receding-horizon control, the same idea as MPC. The model predicts an action chunk of length H, the controller executes only a short prefix of it (say the first k steps, k much less than H), then throws away the rest, observes the resulting frame, and replans from scratch. Predicting long chunks gives temporal coherence and lets the model commit to a multi-step plan like “approach, close, lift.” Executing only a prefix means every few steps the loop is corrected by real observation, so prediction error never compounds. This is the mechanism that turns a generative video model into an actual controller.

  TRAINING STAGE 1 -- action-free video (abundant)
  +---------------------------------------------------+
  |  endoscopic frames o_1..o_t                       |
  |            |                                      |
  |            v                                      |
  |   [ Cosmos Policy backbone / shared latent ]      |
  |            |                                      |
  |            v                                      |
  |   [ video head ] --> o_(t+1)..o_(t+H)             |
  |            ^                                      |
  |            +-- loss: future frame prediction      |
  |  (action head idle -- no labels exist)            |
  +---------------------------------------------------+
                      |
                      | weights carried over
                      v
  TRAINING STAGE 2 -- fixed action-labeled budget (scarce)
  +---------------------------------------------------+
  |  frames o_1..o_t  +  task instruction             |
  |            |                                      |
  |            v                                      |
  |   [ shared latent -- already knows dynamics ]     |
  |          /              \                         |
  |         v                v                        |
  |  [ video head ]     [ action head ]               |
  |   o_(t+1..t+H)       a_(t..t+H) chunk             |
  |         |                |                        |
  |   frame loss        action loss (dVRK kinematics) |
  +---------------------------------------------------+
                      |
                      v
  DEPLOYMENT -- receding horizon closed loop
  +---------------------------------------------------+
  |                                                   |
  |   observe o_t                                     |
  |       |                                           |
  |       v                                           |
  |   predict chunk a_t .. a_(t+H)                    |
  |       |                                           |
  |       v                                           |
  |   execute prefix a_t .. a_(t+k)   [ k << H ]      |
  |       |                                           |
  |       v                                           |
  |   discard tail, observe o_(t+k) ---+              |
  |                                    |              |
  |       +----------------------------+              |
  |       |  replan                                   |
  |       v                                           |
  |   (repeat until task complete)                    |
  +---------------------------------------------------+

The structural metaphor: a surgical resident who watched a thousand operations before ever holding an instrument.

Stage one is the resident in the observation gallery. They watch endoscopic footage for years. Nobody tells them the hand positions or the wrist angles — they have no access to the surgeon’s proprioception. But they absorb something enormous anyway: how tissue springs back, how blood pools, what a needle looks like a half-second before it bites, when a grasper is about to slip. This is pure visual dynamics, learned without a single motor command. That is exactly what action-free video pretraining is.

Stage two is the resident’s first supervised cases at the console. Now they get to move the instruments, and now — crucially — someone is recording what their hands did alongside what the camera saw. They only get a handful of cases; residency is short and cases are precious. But they are not learning surgery from these few cases. They are learning a mapping: “the thing I have watched a thousand times, and can already predict, corresponds to this pattern of wrist and grasper motion.” Learning a mapping from a rich existing model is fast. Learning surgery from twenty cases with no prior observation is not.

Deployment is the resident actually operating. They plan several moves ahead — “come in from the left, close on the peg, lift clear of the board, cross over” — because a plan with a horizon is smoother and safer than one twitch at a time. But they only commit to the first move or two before looking again. Tissue shifts, the peg was not quite where they thought, the endoscope drifted. Look, replan, commit briefly, look again. That is the receding horizon. A resident who executed a full ten-second plan blind would be dangerous; so would a model that ran its whole predicted chunk without re-observing.

And the paper’s headline result is the one every surgical educator already believes: the observation gallery pays off most on the hard stuff. Simple reach-and-place is learnable from a few reps. Contact-rich and bimanual work — where you need a physical intuition of how things push back and how two hands interfere — is where the thousand watched hours show up. PegTransfer gaining 20 points is exactly this pattern.

Key Concepts

• Action-free video pretraining: Normally, teaching a robot means showing it paired data — “this is what the camera saw, and this is what the joints did.” The pairing is the expensive part. Action-free pretraining drops the second half: you show the model only what the camera saw, and ask it to predict what the camera will see next. No robot commands anywhere in the data. Why does this help control at all? Because to predict the next frame well, the model has to internally represent things that matter for acting: where the objects are, that the tool is about to touch something, that the tissue will deform when it does. It builds a physics-flavored understanding of the scene as a side effect of being a good video predictor. Later, when you finally give it a few labeled examples, it doesn’t need to discover that a grasped peg moves with the grasper — it already knows. It just needs to learn which joint numbers correspond to that motion. Concretely: it is the difference between teaching someone to drive who has spent years as a passenger versus someone who has never been in a car.

• Action chunking: The naive way to run a policy is one step at a time: see a frame, output one command, repeat. This has two failure modes — the motion comes out jittery because consecutive decisions are made independently, and the policy can dither indefinitely in ambiguous states because nothing commits it to a plan. Action chunking instead outputs a whole *sequence of, say, 16 commands in one shot. The model has to decide “here is a coherent multi-step motion,” which naturally produces smooth trajectories and forces commitment. The cost is that a chunk is an open-loop plan: if step 3 goes wrong, steps 4 through 16 are predicting a world that no longer exists.

• Receding-horizon execution: This is the fix for chunking’s open-loop problem, borrowed straight from model predictive control. You predict long, execute short. Predict 16 steps; run 4; throw away the other 12; look at the new frame; predict 16 fresh steps. You get chunking’s coherence and commitment, but errors are corrected every 4 steps instead of every 16, so drift cannot accumulate. The mental image is a driver planning the next two hundred meters of road while only actually steering the next twenty, then re-planning. The long plan shapes the short action; the short action keeps the long plan honest.

Framework Shift

Before (mainstream approach):          After (this paper):

TWO SEPARATE PIPELINES                 ONE MODEL, TWO STAGES

  cheap video                            cheap video
      |                                      |
      v                                      v
  [ world model ]                     [ pretrain visual dynamics ]
      |                                      |
      v                                      v  (same weights)
  predicted frames                    [ + action head, fine-tune ]
      |                                      ^
      v                                      |
  ( watch it / use as sim )            scarce action labels
  ( eval a policy against it )                |
                                              v
   ...dead end for control...          [ predict frames AND
                                          action chunks ]
  ------------------------------              |
                                              v
  scarce action labels                 execute prefix, replan
      |                                       |
      v                                       v
  [ imitation policy ]                 CLOSED LOOP ON ROBOT
      |
      v
  closed loop, but                     video is not a simulator,
  data-starved, fails on               it is a source of
  contact-rich + bimanual              transferable priors

  video and control never
  share a representation

One sentence: from video as a simulator you evaluate policies against to video as unlabeled pretraining data whose learned dynamics flow directly into the policy’s own weights, the core shift is treating endoscopic footage as a representation-learning corpus for control rather than a rendering target.

Expert Assessment

Problem choice: Real gap, and unusually well-posed. The scarcity of synchronized video-kinematics data in surgical robotics is not a manufactured constraint — it is the actual reason surgical imitation learning lags behind general manipulation. And the framing is disciplined: *under a fixed budget of action labels, does action-free video help? That phrasing rules out the cheap win of “we used more data and got better numbers.” Where it sits in the trajectory: this is the surgical instantiation of a broader move that GR-1/GR-2, UniPi, and the Cosmos line have been pushing in general robotics. The paper is not inventing the world-action-model idea; it is running the domain-specific experiment that the surgical community needed someone to run. That is a legitimate contribution, but it is an application-and-validation contribution, not a conceptual one, and the abstract could be more upfront about that.

Method maturity: This is clean engineering rather than a clever new mechanism. Cosmos Policy already gives you the joint video-plus-action architecture; the contributions are the surgical pretraining corpus, the two-stage recipe, and the receding-horizon deployment loop. All three are sensible and none are surprising. What I’d want to see argued against: cheaper baselines that might capture much of the gain. Plain visual representation pretraining — a frozen video encoder, R3M/VC-1-style, or a masked-autoencoder on endoscopic frames feeding a small ACT head — is far less compute and might get you most of the way. Latent-action approaches (LAPO, Genie-style inverse dynamics in latent space) are a closer competitor and would test whether *generative future prediction specifically is what matters, versus just good features. If those comparisons are absent, the paper shows that its pipeline works but not that the expensive part is doing the work.

Experimental integrity: The headline structure is right — fixed label budget, same architecture, pretraining as the single toggled variable. That is the correct ablation and it deserves credit. But four simulated tasks is thin, and everything here is simulation. Surgical robotics has a specific credibility problem: endoscopic sim-to-real is brutal (specular highlights, deformable tissue, smoke, blood, lighting), and the paper’s central claim is about visual dynamics priors — exactly the thing most likely to be sim-flavored. If the pretraining video is also simulated or drawn from the same simulator distribution, the result is much weaker than it reads: it becomes near-domain transfer rather than “cheap real video helps.” The abstract does not say, and that ambiguity is the single biggest thing I’d want resolved. Also: 63.5% to 77.8% averaged over four tasks means each task contributes ~25% of the average, so a +20-point swing on PegTransfer alone accounts for a third of the total gain. I want per-task numbers, seed variance, and confidence intervals before treating 77.8% as a stable figure. With four tasks and presumably tens of eval episodes each, the error bars could plausibly be ±5 points or worse. No dVRK hardware results is the honest limitation, and framing this as “a practical path toward scaling up surgical robot learning” is a stretch until something moves on real hardware.

Writing quality: The abstract is well-constructed and states a falsifiable question, which is more than most papers manage. The corner-cutting is almost certainly in the data description — the paper’s entire argument rests on what the action-free video corpus actually is (source, scale, domain gap to the eval tasks), and abstracts that stay vague there usually do so because the answer is less impressive than the framing. Rewriting the data and experimental-setup section with full transparency on corpus provenance, size, per-task breakdowns, and seed variance would elevate the whole paper from “plausible claim” to “settled result.” Second priority: a scaling curve. How does the gain vary with the action-label budget, and how does it vary with pretraining hours? Those two curves would turn this from a single data point into a usable finding others could plan around.

Verdict: weak accept — the question is right, the ablation is structured correctly, and the result is directionally valuable, but simulation-only evidence on four tasks with likely-missing cheaper baselines makes it a solid domain validation rather than a definitive one.

Takeaways

Things a practitioner can actually lift from this:

• The fixed-budget framing is a template. Whenever you claim auxiliary data helps, hold the expensive supervision constant and toggle only the cheap data. It kills the confound that makes most “pretraining helps” claims uninterpretable. Steal this experimental design regardless of domain.

• Predict long, execute short. The chunk-length-versus-execution-length gap is a free tuning knob that most people leave at 1

  . Long chunks buy temporal coherence and commitment; short execution prefixes buy error correction. If your imitation policy is jittery or dithers in ambiguous states, this ratio is the first thing to tune, and it costs nothing to try.

• Video prediction as a proxy objective for physical understanding. The transferable insight is not “use Cosmos” — it is that forcing a model to predict pixels makes it encode contact, occlusion, and object permanence for free. Anywhere you have abundant unlabeled sensor streams and scarce action labels — industrial inspection, agricultural robotics, lab automation — the same two-stage recipe applies.

• The gain concentrates on contact-rich and bimanual tasks. This is the most decision-relevant number in the paper. If your tasks are free-space reaching, video pretraining probably isn’t worth the compute. If they involve contact dynamics or coordinated multi-arm motion, that’s where the prior pays. Use this to triage whether to invest.

• Read the data section before believing the result. Meta-lesson: for any paper claiming cheap-data-helps-expensive-task, the whole claim lives or dies on the domain gap between the cheap corpus and the eval environment. If a paper is vague about corpus provenance, treat the number as an upper bound.