SurgicalGaussian: Deformable 3D Gaussians for High-Fidelity Surgical Scene Reconstruction

TL;DR

SurgicalGaussian introduces a deformable 3D Gaussian Splatting method for high-fidelity, real-time reconstruction of dynamic surgical scenes, effectively handling occlusions and intricate tissue details in endoscopic videos.

Contribution

It presents a novel deformable 3D Gaussian approach that models soft tissue dynamics and improves reconstruction quality and speed over existing neural radiance field-based methods.

Findings

Outperforms existing methods in rendering quality

Achieves faster rendering speeds

Uses less GPU memory

Abstract

Dynamic reconstruction of deformable tissues in endoscopic video is a key technology for robot-assisted surgery. Recent reconstruction methods based on neural radiance fields (NeRFs) have achieved remarkable results in the reconstruction of surgical scenes. However, based on implicit representation, NeRFs struggle to capture the intricate details of objects in the scene and cannot achieve real-time rendering. In addition, restricted single view perception and occluded instruments also propose special challenges in surgical scene reconstruction. To address these issues, we develop SurgicalGaussian, a deformable 3D Gaussian Splatting method to model dynamic surgical scenes. Our approach models the spatio-temporal features of soft tissues at each time stamp via a forward-mapping deformation MLP and regularization to constrain local 3D Gaussians to comply with consistent movement. With the depth initialization strategy and tool mask-guided training, our method can remove surgical instruments and reconstruct high-fidelity surgical scenes. Through experiments on various surgical videos, our network outperforms existing method on many aspects, including rendering quality, rendering speed and GPU usage. The project page can be found at https://surgicalgaussian.github.io.