SurgCUT3R: Surgical Scene-Aware Continuous Understanding of Temporal 3D Representation

TL;DR

SurgCUT3R introduces a domain-adapted 3D reconstruction framework for surgical scenes from monocular videos, addressing data scarcity and long-sequence accuracy issues with novel data generation, hybrid supervision, and hierarchical inference.

Contribution

It presents a comprehensive framework combining data synthesis, hybrid supervision, and hierarchical inference to improve surgical scene reconstruction from monocular videos.

Findings

Achieves near state-of-the-art accuracy with higher efficiency.

Effectively mitigates pose drift over long surgical videos.

Demonstrates robustness on multiple surgical datasets.

Abstract

Reconstructing surgical scenes from monocular endoscopic video is critical for advancing robotic-assisted surgery. However, the application of state-of-the-art general-purpose reconstruction models is constrained by two key challenges: the lack of supervised training data and performance degradation over long video sequences. To overcome these limitations, we propose SurgCUT3R, a systematic framework that adapts unified 3D reconstruction models to the surgical domain. Our contributions are threefold. First, we develop a data generation pipeline that exploits public stereo surgical datasets to produce large-scale, metric-scale pseudo-ground-truth depth maps, effectively bridging the data gap. Second, we propose a hybrid supervision strategy that couples our pseudo-ground-truth with geometric self-correction to enhance robustness against inherent data imperfections. Third, we introduce a hierarchical inference framework that employs two specialized models to effectively mitigate accumulated pose drift over long surgical videos: one for global stability and one for local accuracy. Experiments on the SCARED and StereoMIS datasets demonstrate that our method achieves a competitive balance between accuracy and efficiency, delivering near state-of-the-art but substantially faster pose estimation and offering a practical and effective solution for robust reconstruction in surgical environments. Project page: https://chumo-xu.github.io/SurgCUT3R-ICRA26/.