ColorGS: High-fidelity Surgical Scene Reconstruction with Colored Gaussian Splatting

TL;DR

ColorGS introduces a novel surgical scene reconstruction framework that combines adaptive color encoding and advanced deformation modeling to achieve high-fidelity, real-time 3D reconstructions from endoscopic videos, surpassing previous methods.

Contribution

The paper presents Colored Gaussian Primitives with learnable color parameters and an Enhanced Deformation Model integrating local and global tissue deformations, advancing the state-of-the-art in surgical scene reconstruction.

Findings

Achieves a PSNR of 39.85, 1.5 higher than prior methods.

Attains SSIM of 97.25%, demonstrating high structural similarity.

Maintains real-time rendering efficiency for intraoperative use.

Abstract

High-fidelity reconstruction of deformable tissues from endoscopic videos remains challenging due to the limitations of existing methods in capturing subtle color variations and modeling global deformations. While 3D Gaussian Splatting (3DGS) enables efficient dynamic reconstruction, its fixed per-Gaussian color assignment struggles with intricate textures, and linear deformation modeling fails to model consistent global deformation. To address these issues, we propose ColorGS, a novel framework that integrates spatially adaptive color encoding and enhanced deformation modeling for surgical scene reconstruction. First, we introduce Colored Gaussian Primitives, which employ dynamic anchors with learnable color parameters to adaptively encode spatially varying textures, significantly improving color expressiveness under complex lighting and tissue similarity. Second, we design an Enhanced Deformation Model (EDM) that combines time-aware Gaussian basis functions with learnable time-independent deformations, enabling precise capture of both localized tissue deformations and global motion consistency caused by surgical interactions. Extensive experiments on DaVinci robotic surgery videos and benchmark datasets (EndoNeRF, StereoMIS) demonstrate that ColorGS achieves state-of-the-art performance, attaining a PSNR of 39.85 (1.5 higher than prior 3DGS-based methods) and superior SSIM (97.25\%) while maintaining real-time rendering efficiency. Our work advances surgical scene reconstruction by balancing high fidelity with computational practicality, critical for intraoperative guidance and AR/VR applications.