4D Monocular Surgical Reconstruction under Arbitrary Camera Motions

TL;DR

This paper introduces Local-EndoGS, a novel 4D reconstruction framework for monocular endoscopic videos with arbitrary camera motion, overcoming initialization challenges and improving scene reconstruction quality in clinical settings.

Contribution

The paper presents a scalable, window-based 4D reconstruction method that integrates multi-view geometry and monocular priors, enabling robust deformable scene reconstruction with large camera motions.

Findings

Outperforms state-of-the-art methods in appearance and geometry quality.

Effective in handling long sequences with substantial camera motion.

Validated on three public endoscopic datasets.

Abstract

Reconstructing deformable surgical scenes from endoscopic videos is challenging and clinically important. Recent state-of-the-art methods based on implicit neural representations or 3D Gaussian splatting have made notable progress. However, most are designed for deformable scenes with fixed endoscope viewpoints and rely on stereo depth priors or accurate structure-from-motion for initialization and optimization, limiting their ability to handle monocular sequences with large camera motion in real clinical settings. To address this, we propose Local-EndoGS, a high-quality 4D reconstruction framework for monocular endoscopic sequences with arbitrary camera motion. Local-EndoGS introduces a progressive, window-based global representation that allocates local deformable scene models to each observed window, enabling scalability to long sequences with substantial motion. To overcome unreliable initialization without stereo depth or accurate structure-from-motion, we design a coarse-to-fine strategy integrating multi-view geometry, cross-window information, and monocular depth priors, providing a robust foundation for optimization. We further incorporate long-range 2D pixel trajectory constraints and physical motion priors to improve deformation plausibility. Experiments on three public endoscopic datasets with deformable scenes and varying camera motions show that Local-EndoGS consistently outperforms state-of-the-art methods in appearance quality and geometry. Ablation studies validate the effectiveness of our key designs. Code will be released upon acceptance at: https://github.com/IRMVLab/Local-EndoGS.