High-fidelity Endoscopic Image Synthesis by Utilizing Depth-guided Neural Surfaces

TL;DR

This paper presents a novel method for high-fidelity colon section reconstruction using Neural Implicit Surfaces with a single depth map, improving accuracy and scale stability in endoscopic image synthesis.

Contribution

Introducing a pioneering approach that leverages NeuS with only one depth frame for photorealistic colon reconstruction, addressing texture and scale issues in monocular endoscopic imaging.

Findings

Achieves highly accurate colon surface rendering

Captures unseen surface portions effectively

Demonstrates robustness with phantom imagery

Abstract

In surgical oncology, screening colonoscopy plays a pivotal role in providing diagnostic assistance, such as biopsy, and facilitating surgical navigation, particularly in polyp detection. Computer-assisted endoscopic surgery has recently gained attention and amalgamated various 3D computer vision techniques, including camera localization, depth estimation, surface reconstruction, etc. Neural Radiance Fields (NeRFs) and Neural Implicit Surfaces (NeuS) have emerged as promising methodologies for deriving accurate 3D surface models from sets of registered images, addressing the limitations of existing colon reconstruction approaches stemming from constrained camera movement. However, the inadequate tissue texture representation and confused scale problem in monocular colonoscopic image reconstruction still impede the progress of the final rendering results. In this paper, we introduce a novel method for colon section reconstruction by leveraging NeuS applied to endoscopic images, supplemented by a single frame of depth map. Notably, we pioneered the exploration of utilizing only one frame depth map in photorealistic reconstruction and neural rendering applications while this single depth map can be easily obtainable from other monocular depth estimation networks with an object scale. Through rigorous experimentation and validation on phantom imagery, our approach demonstrates exceptional accuracy in completely rendering colon sections, even capturing unseen portions of the surface. This breakthrough opens avenues for achieving stable and consistently scaled reconstructions, promising enhanced quality in cancer screening procedures and treatment interventions.