SLAM Endoscopy enhanced by adversarial depth prediction

TL;DR

This paper introduces a novel SLAM method for endoscopy that uses adversarially-trained CNNs for depth prediction, enabling dense scene reconstruction despite the lack of direct depth sensing.

Contribution

It presents a new approach combining adversarial depth prediction with SLAM for endoscopic imaging, trained on synthetic and domain-randomized images for improved accuracy.

Findings

Enables dense reconstruction of endoscopic scenes

Uses adversarial training for monocular depth prediction

Improves SLAM performance in medical endoscopy

Abstract

Medical endoscopy remains a challenging application for simultaneous localization and mapping (SLAM) due to the sparsity of image features and size constraints that prevent direct depth-sensing. We present a SLAM approach that incorporates depth predictions made by an adversarially-trained convolutional neural network (CNN) applied to monocular endoscopy images. The depth network is trained with synthetic images of a simple colon model, and then fine-tuned with domain-randomized, photorealistic images rendered from computed tomography measurements of human colons. Each image is paired with an error-free depth map for supervised adversarial learning. Monocular RGB images are then fused with corresponding depth predictions, enabling dense reconstruction and mosaicing as an endoscope is advanced through the gastrointestinal tract. Our preliminary results demonstrate that incorporating monocular depth estimation into a SLAM architecture can enable dense reconstruction of endoscopic scenes.