PR-ENDO: Physically Based Relightable Gaussian Splatting for Endoscopy

TL;DR

PR-ENDO introduces a physically based relightable Gaussian Splatting framework for endoscopic 3D reconstruction, improving accuracy, generalization, and enabling tissue modifications for clinical applications.

Contribution

It presents a novel endoscopic 3D reconstruction method that separates light effects from tissue properties using a relightable Gaussian Splatting model with a specialized MLP.

Findings

Achieves superior reconstruction quality over baseline methods.

Enables tissue modifications while maintaining physically accurate lighting response.

Reduces artifacts and improves generalization across views.

Abstract

Endoluminal endoscopic procedures are essential for diagnosing colorectal cancer and other severe conditions in the digestive tract, urogenital system, and airways. 3D reconstruction and novel-view synthesis from endoscopic images are promising tools for enhancing diagnosis. Moreover, integrating physiological deformations and interaction with the endoscope enables the development of simulation tools from real video data. However, constrained camera trajectories and view-dependent lighting create artifacts, leading to inaccurate or overfitted reconstructions. We present PR-ENDO, a novel 3D reconstruction framework leveraging the unique property of endoscopic imaging, where a single light source is closely aligned with the camera. Our method separates light effects from tissue properties. PR-ENDO enhances 3D Gaussian Splatting with a physically based relightable model. We boost the traditional light transport formulation with a specialized MLP capturing complex light-related effects while ensuring reduced artifacts and better generalization across novel views. PR-ENDO achieves superior reconstruction quality compared to baseline methods on both public and in-house datasets. Unlike existing approaches, PR-ENDO enables tissue modifications while preserving a physically accurate response to light, making it closer to real-world clinical use.