Stop Holding Your Breath: CT-Informed Gaussian Splatting for Dynamic Bronchoscopy

Paper: 2604.28179 Authors: Andrea Dunn Beltran, Daniel Rho, Aarav Mehta, Xinqi Xiong, Raúl San José Estépar, Ron Alterovitz, Marc Niethammer, Roni Sengupta Categories: cs.CV

The Gap

Bronchoscopic navigation systems register live endoscopic video to a preoperative CT scan to guide physicians through the airway. The problem: breathing deforms the airway by 5-20mm, creating a mismatch between the static CT and the moving anatomy. Current solutions force patients to hold their breath during the procedure, attempting to freeze the anatomy to match the CT. But breath-holds are hard to reproduce consistently, disrupt workflow, and still don’t guarantee the anatomy matches the CT’s breathing phase.

Prior work in endoscopic reconstruction (NeRF-based methods, standard Gaussian splatting) treats each frame independently or assumes static scenes. They can’t handle respiratory motion without external sensors or multiple synchronized cameras. The clinical workaround—breath-hold protocols—is a band-aid that shifts the burden to the patient rather than solving the underlying CT-to-body divergence problem.

Problem: Breathing deforms airway 5-20mm
    |
    v
Current fix: Force breath-holds (unreliable, disruptive)
    |
    v
Root cause: Static CT vs dynamic anatomy mismatch
    |
    v
Key insight: Paired inhale/exhale CTs define patient-specific deformation space
    |
    v
Method: Constrain reconstruction to anatomically observed configurations
    |
    v
Evidence: 1.22mm localization (< 3mm clinical threshold), 20x faster training
    |
    v
Conclusion: Breath-holds unnecessary when deformation is modeled correctly

The Increment

One sentence: Before this paper, bronchoscopy required breath-holds to match static CT scans; after, the system adapts to natural breathing by learning patient-specific respiratory motion from routine planning scans.

Core Mechanism

The method starts with paired inhale-exhale CT scans already acquired during surgical planning. These two scans implicitly define the patient’s respiratory deformation space—the range of anatomically plausible airway configurations. The authors register these CTs to extract a deformation field, then reduce the entire breathing cycle to a single scalar: breathing phase (0 = full exhale, 1 = full inhale).

During bronchoscopy, a lightweight neural network estimates breathing phase directly from the endoscopic RGB frame. This phase value drives a mesh-anchored Gaussian splatting framework. The Gaussians are attached to a tetrahedral mesh of the airway, and the mesh deforms according to the estimated breathing phase, interpolating between the registered inhale and exhale states. This constrains all reconstructions to lie within the anatomically observed deformation space—no hallucinated geometry, no drift into physically impossible configurations.

Preoperative:
  CT_inhale + CT_exhale  -->  [Registration]  -->  Deformation field
                                                           |
                                                           v
Intraoperative:                                    Phase interpolation
  RGB frame  -->  [Phase estimator]  -->  phase (0 to 1)  |
                                                           v
                                          [Mesh deformation]
                                                           |
                                                           v
                                          Deformed Gaussians  -->  Render

Think of it like a flip book with only two drawings: full inhale and full exhale. Instead of drawing every intermediate frame from scratch (which would be guessing), you morph smoothly between the two known drawings. The breathing phase is just the page number—0% through the flip book is exhale, 100% is inhale, 50% is halfway between. The endoscope camera watches the airway and estimates which page you’re on in real-time. Because you’re always interpolating between two real anatomical snapshots, you never draw something that couldn’t physically exist in that patient’s airway.

Key Concepts

• Breathing phase as a constraint: Most reconstruction methods treat each video frame as an independent observation and try to build geometry from scratch. This paper collapses the entire respiratory cycle into a single number (0 to 1) that indexes into a pre-registered deformation space. It’s not estimating arbitrary 3D motion—it’s estimating where you are along a known 1D path. This massively reduces the degrees of freedom and prevents the reconstruction from drifting into anatomically impossible states. Imagine trying to navigate a city with a full map versus being told “you’re 37% of the way between the train station and the park”—the second gives you far less freedom to get lost.

• Mesh-anchored Gaussians: Standard Gaussian splatting places 3D Gaussians freely in space. Here, Gaussians are attached to vertices of a tetrahedral mesh that represents the airway. When the mesh deforms (driven by breathing phase), the Gaussians move with it. This coupling ensures geometric consistency—Gaussians can’t float away or cluster in weird configurations because they’re tethered to an anatomically meaningful structure. It’s like hanging Christmas lights on a tree branch: the lights follow the branch’s shape, and if the branch bends, the lights bend with it.

• RESPIRE simulation pipeline: Evaluating bronchoscopy methods is hard because you can’t get ground truth for camera pose, breathing phase, and deformation in real procedures. RESPIRE is a physics-based simulator that generates synthetic bronchoscopy videos with per-frame ground truth for everything. It uses real CT scans, simulates realistic camera motion and lighting, and provides the labels needed for quantitative evaluation. Without this, the field would be stuck with qualitative assessments or proxy metrics that don’t directly measure what matters clinically.

Framework Shift

Before (mainstream approach):        After (this paper):

Static CT scan                       Paired CT scans (inhale + exhale)
      |                                      |
      v                                      v
Breath-hold protocol              Deformation field extraction
      |                                      |
      v                                      v
Match anatomy to CT               Estimate breathing phase from RGB
      |                                      |
      v                                      v
Reconstruct from video            Interpolate within deformation space
      |                                      |
      v                                      v
Hope anatomy stayed still         Adapt to natural breathing

From forcing the patient to match the CT, to making the CT model match the patient—the core shift is moving the adaptation burden from the human to the algorithm.

Expert Assessment

Problem choice: This is a real clinical pain point, not a manufactured gap. Breath-hold protocols are genuinely disruptive and unreliable. The paper sits at a sweet intersection: leveraging data already collected (paired CTs are standard for planning) and addressing a workflow bottleneck. The timing is right—Gaussian splatting is mature enough to be a stable foundation, and the medical imaging community is hungry for methods that reduce patient burden.

Method maturity: The core insight—using paired CTs to constrain the deformation space—is elegant and well-motivated. The execution is solid: mesh-anchored Gaussians are a natural fit, and the phase estimator is lightweight enough to be practical. However, the method assumes the deformation is well-approximated by linear interpolation between two extremes, which may not hold for complex breathing patterns or coughing. The paper doesn’t explore failure modes when the assumption breaks. Also, the reliance on tetrahedral mesh quality could be a hidden dependency—bad meshing could degrade results, but this isn’t discussed.

Experimental integrity: The RESPIRE simulator is a major contribution and the experiments are thorough within its scope. The 1.22mm localization accuracy is impressive and clinically relevant. However, all quantitative results are on simulated data. The real bronchoscopy examples are qualitative only. This is understandable (ground truth is hard), but it means we don’t know how the method degrades with real-world noise, mucus, bleeding, or atypical anatomy. The baselines (single-CT methods) are fair, but there’s no comparison to other respiratory motion modeling approaches from the broader medical imaging literature.

Writing quality: The paper is well-structured and the motivation is crystal clear. The related work section is thorough. However, the method section could be more explicit about failure modes and assumptions. The discussion of mesh quality and interpolation limits is buried or absent. The results section leans heavily on the simulator—adding even one quantitative metric from real data (e.g., reprojection error on a small annotated set) would strengthen confidence. The supplementary material and website are excellent.

Verdict: weak accept — Solid clinical motivation, elegant method, and strong simulator-based results, but lacks real-world quantitative validation and deeper analysis of when the linear interpolation assumption fails.

Takeaways

The big transferable idea: when you have paired observations of a system’s extremes (min/max, start/end, before/after), you can constrain a dynamic reconstruction to interpolate between them rather than solving the full inverse problem from scratch. This reduces degrees of freedom and prevents drift. It’s applicable beyond bronchoscopy—think cardiac imaging (systole/diastole), musculoskeletal motion (flexion/extension), or even non-medical domains like deformable object manipulation where you can capture boundary states.

The mesh-anchored Gaussian trick is also worth stealing: if you’re reconstructing something with known topology, tether your representation to a mesh rather than letting primitives float freely. It’s a simple way to inject structural priors without heavy regularization.

Finally, the RESPIRE simulator is a template for how to build evaluation infrastructure when ground truth is inaccessible. Physics-based simulation with real data inputs (CT scans) bridges the gap between synthetic benchmarks and clinical reality better than purely synthetic scenes.