Teaching Surgical AI to Think Like a Surgeon: Mining Reasoning from Video Lectures

Paper: 2603.06570 Authors: Alejandra Perez, Anita Rau, Lee White, Busisiwe Mlambo, Chinedu Nwoye, Muhammad Abdullah Jamal, Omid Mohareri Categories: cs.CV, cs.AI

The Gap

Surgical AI has gotten good at the “what”—recognizing instruments, tracking phases, segmenting anatomy. Models like CholecT50 and EndoVis can label surgical scenes with decent accuracy. But ask them “why is the surgeon using monopolar scissors here instead of bipolar?” or “what complication risk does this tissue manipulation pose?” and they go silent. The problem isn’t model architecture—it’s training data. Existing surgical datasets (CholecT45, PSI-AVA, Surgical-VQLA) provide labels for actions and objects, but not the reasoning behind them. Annotating surgical rationale at scale requires expert surgeons to watch thousands of hours of video and articulate their thought process for each decision—economically infeasible.

Yet the reasoning already exists, narrated aloud in surgical teaching videos. When an expert records a lecture, they explain “I’m choosing this approach because…” and “watch for bleeding here due to…” This supervision is noisy (mixed with tangential commentary) and unstructured (free-form speech), but it encodes exactly what’s missing: intent, risk assessment, and anticipation.

Problem: Surgical AI lacks reasoning ability
   |
   v
Observation: Teaching videos contain expert reasoning (noisy, unstructured)
   |
   v
Method: Multi-agent pipeline extracts QA pairs from lecture narrations
   |
   v
Evidence: 206.8K QA pairs across 12 reasoning categories, models reach 84% accuracy
   |
   v
Conclusion: Harvesting teaching signal enables surgical reasoning without manual annotation

The Increment

One sentence: Before SUREON, surgical AI could recognize what’s happening in an OR; after SUREON, it can explain why it’s happening and what might happen next.

Core Mechanism

SUREON’s pipeline has three stages. First, it ingests surgical teaching videos (YouTube lectures, recorded procedures with expert commentary) and uses speech recognition to extract narration transcripts. Second, a multi-agent LLM system processes these transcripts: one agent identifies reasoning-rich segments (explanations of technique choice, risk warnings, outcome predictions), another generates question-answer pairs following 12 predefined categories (safety assessment, decision rationale, forecasting, etc.), and a third validates coherence and grounds answers in visual content. Third, the system pairs each QA with its corresponding video clip, creating training triplets of (video, question, answer).

[Video Lecture] --speech-to-text--> [Transcript]
                                         |
                                         v
                              +----------+----------+
                              |  Multi-Agent LLM   |
                              +----------+----------+
                                         |
                    +--------------------+--------------------+
                    |                    |                    |
                    v                    v                    v
              [Identifier]          [Generator]         [Validator]
              (finds reasoning)   (creates QA pairs)   (checks quality)
                    |                    |                    |
                    +--------------------+--------------------+
                                         |
                                         v
                              [QA Pair + Video Clip]
                                         |
                                         v
                              [Training Dataset: 206.8K pairs]

Think of this like a medical school study group that records itself. The recording captures not just the procedure demonstration, but the running commentary: “I’m using this suture because…”, “if you see this sign, worry about…”. SUREON is the diligent student who rewatches all recordings, extracts every teaching moment, and organizes them into flashcards. The “identifier agent” is the student marking timestamps where the instructor explains something important. The “generator agent” turns those explanations into quiz questions. The “validator agent” is the study partner who checks if the flashcard makes sense and actually relates to what’s shown on screen. The result: a massive deck of surgical reasoning flashcards, automatically generated from existing teaching materials.

Key Concepts

• Reasoning vs. Perception in Surgical AI: Perception means labeling what you see—“that’s a grasper, that’s the gallbladder”. Reasoning means understanding the clinical logic—“the surgeon chose a grasper instead of scissors because blunt dissection reduces bleeding risk in this inflamed tissue”. Current surgical datasets provide perception labels (bounding boxes, phase annotations). SUREON provides reasoning supervision (why this tool, what could go wrong, what happens next). It’s the difference between a medical student who can name anatomy and one who can explain a treatment plan.

• Multi-Agent Extraction Pipeline: Instead of one LLM doing everything, SUREON uses specialized agents in sequence. Why? Because extracting reasoning from noisy lecture transcripts requires different skills: finding relevant segments (filtering), generating coherent questions (creativity), and validating visual grounding (quality control). One agent might hallucinate, but three agents checking each other’s work reduces errors. It’s like having a writer, an editor, and a fact-checker instead of one person doing all three jobs poorly.

• Group Relative Policy Optimization (GRPO): Standard supervised fine-tuning teaches a model to mimic training examples. GRPO goes further—it trains the model to generate reasoning chains (like chain-of-thought), then rewards outputs that lead to correct answers. The model learns not just to answer, but to show its work. Imagine teaching a student by grading not just their final answer, but the quality of their reasoning steps. GRPO does this by comparing multiple reasoning attempts and reinforcing the better ones.

Framework Shift

Before (mainstream approach):        After (this paper):

[Surgical Video]                     [Surgical Video + Lecture Audio]
       |                                      |
       v                                      v
[Manual Annotation]                  [Multi-Agent Extraction]
 (experts label actions)              (LLMs mine reasoning)
       |                                      |
       v                                      v
[Perception Dataset]                 [Reasoning Dataset]
 (what is happening)                  (why + what if)
       |                                      |
       v                                      v
[Recognition Model]                  [Reasoning Model]
 "This is a grasper"                  "Grasper chosen for blunt
                                       dissection to reduce
                                       bleeding risk"

From manually annotating what’s visible to automatically harvesting why it matters, the core shift is treating teaching narration as free supervision instead of noise to be discarded.

Expert Assessment

Problem choice: This is a real gap. Surgical AI has plateaued at perception tasks—adding more labeled instruments won’t make models clinically useful. The leap to reasoning is necessary for deployment, and the observation that teaching videos already contain this signal is sharp. It sits at the frontier where computer vision meets clinical decision support.

Method maturity: The multi-agent pipeline is clever but not novel—it’s competent engineering of existing LLM techniques. The real contribution is problem framing: recognizing that lecture narrations are harvestable supervision. GRPO for surgical reasoning is a reasonable choice, though the paper doesn’t deeply explore why it outperforms alternatives. A simpler approach might be direct prompting of large VLMs, but the authors don’t test this baseline rigorously.

Experimental integrity: The benchmark (354 expert-validated examples) is small but carefully curated. Baselines include GPT-4V and Gemini, which is fair. The 84% accuracy sounds impressive until you realize the benchmark might be easier than real clinical scenarios—questions are derived from the same lecture corpus used for training, risking distribution overlap. The paper acknowledges this but doesn’t quantify the risk. Ablations are present but shallow—no analysis of which question categories benefit most from reasoning vs. supervised fine-tuning.

Writing quality: The abstract and introduction are strong. The method section buries important details (how are the 12 question categories defined? what prompts guide the agents?) in supplementary material. The results section focuses on aggregate accuracy without enough error analysis—which reasoning types still fail? The qualitative examples are cherry-picked; showing failure cases would build trust.

Verdict: Weak accept—the dataset contribution is valuable and the problem framing is important, but the modeling work feels incremental and the evaluation could be more rigorous.

Takeaways

The transferable idea: when expert reasoning is too expensive to annotate but already exists in unstructured teaching materials (lectures, tutorials, documentation), use multi-agent LLM pipelines to extract and structure it at scale. This applies beyond surgery—think code review comments for programming education, design critiques for architecture training, or strategy explanations in game replays. The key is identifying domains where experts naturally narrate their reasoning for teaching purposes, then systematically harvesting that signal.

Specific technique: the three-agent pattern (identifier → generator → validator) is a practical recipe for quality control when mining noisy text. Don’t trust one LLM to do extraction, generation, and validation—specialize and sequence them.

Warning: the paper doesn’t address a critical deployment question—how do you prevent the model from confidently hallucinating reasoning that sounds plausible but is clinically wrong? Surgical errors kill people. The 84% accuracy means 16% failure rate, which is unacceptable in real ORs. The paper treats this as a research benchmark problem, not a clinical deployment problem, which is honest but limits immediate applicability.