NeRFscopy: Neural Radiance Fields for in-vivo Time-Varying Tissues from Endoscopy

TL;DR

NeRFscopy is a self-supervised neural rendering pipeline that enables accurate 3D reconstruction and novel view synthesis of deformable tissues in endoscopic videos, overcoming challenges like tissue deformation and camera motion.

Contribution

It introduces a deformable neural radiance field model with a canonical and time-dependent deformation, tailored for endoscopic tissue reconstruction from monocular videos without pre-training.

Findings

Outperforms existing methods in view synthesis accuracy

Handles tissue deformation and camera motion effectively

Does not require pre-trained models or templates

Abstract

Endoscopy is essential in medical imaging, used for diagnosis, prognosis and treatment. Developing a robust dynamic 3D reconstruction pipeline for endoscopic videos could enhance visualization, improve diagnostic accuracy, aid in treatment planning, and guide surgery procedures. However, challenges arise due to the deformable nature of the tissues, the use of monocular cameras, illumination changes, occlusions and unknown camera trajectories. Inspired by neural rendering, we introduce NeRFscopy, a self-supervised pipeline for novel view synthesis and 3D reconstruction of deformable endoscopic tissues from a monocular video. NeRFscopy includes a deformable model with a canonical radiance field and a time-dependent deformation field parameterized by SE(3) transformations. In addition, the color images are efficiently exploited by introducing sophisticated terms to learn a 3D implicit model without assuming any template or pre-trained model, solely from data. NeRFscopy achieves accurate results in terms of novel view synthesis, outperforming competing methods across various challenging endoscopy scenes.