LightNeuS: Neural Surface Reconstruction in Endoscopy using Illumination Decline

TL;DR

LightNeuS introduces a novel neural surface reconstruction method for endoscopy that models scene illumination decay, enabling watertight 3D reconstructions of colon sections and potential for automatic cancer screening assessment.

Contribution

It extends NeuS by incorporating illumination decline and calibrated photometric modeling, enabling watertight endoscopic surface reconstructions with variable lighting conditions.

Findings

Achieves accurate watertight reconstructions of colon sections.

Allows completion of unseen surface portions with acceptable accuracy.

Demonstrates effectiveness on phantom imagery.

Abstract

We propose a new approach to 3D reconstruction from sequences of images acquired by monocular endoscopes. It is based on two key insights. First, endoluminal cavities are watertight, a property naturally enforced by modeling them in terms of a signed distance function. Second, the scene illumination is variable. It comes from the endoscope's light sources and decays with the inverse of the squared distance to the surface. To exploit these insights, we build on NeuS, a neural implicit surface reconstruction technique with an outstanding capability to learn appearance and a SDF surface model from multiple views, but currently limited to scenes with static illumination. To remove this limitation and exploit the relation between pixel brightness and depth, we modify the NeuS architecture to explicitly account for it and introduce a calibrated photometric model of the endoscope's camera and light source. Our method is the first one to produce watertight reconstructions of whole colon sections. We demonstrate excellent accuracy on phantom imagery. Remarkably, the watertight prior combined with illumination decline, allows to complete the reconstruction of unseen portions of the surface with acceptable accuracy, paving the way to automatic quality assessment of cancer screening explorations, measuring the global percentage of observed mucosa.