V$^2$-SfMLearner: Learning Monocular Depth and Ego-motion for Multimodal Wireless Capsule Endoscopy

TL;DR

V$^2$-SfMLearner is a novel multimodal deep learning approach that integrates vibration signals with visual data to improve depth and ego-motion estimation in monocular capsule endoscopy, enhancing robustness and clinical applicability.

Contribution

It introduces a multimodal unsupervised learning framework incorporating vibration signals, with a specialized vibration network and Fourier fusion module, to improve depth and motion estimation in capsule endoscopy.

Findings

Outperforms vision-only algorithms in accuracy and robustness.

Effectively mitigates vibration noise through multimodal fusion.

Compatible with existing vision-based methods for clinical use.

Abstract

Deep learning can predict depth maps and capsule ego-motion from capsule endoscopy videos, aiding in 3D scene reconstruction and lesion localization. However, the collisions of the capsule endoscopies within the gastrointestinal tract cause vibration perturbations in the training data. Existing solutions focus solely on vision-based processing, neglecting other auxiliary signals like vibrations that could reduce noise and improve performance. Therefore, we propose V$^2$-SfMLearner, a multimodal approach integrating vibration signals into vision-based depth and capsule motion estimation for monocular capsule endoscopy. We construct a multimodal capsule endoscopy dataset containing vibration and visual signals, and our artificial intelligence solution develops an unsupervised method using vision-vibration signals, effectively eliminating vibration perturbations through multimodal learning. Specifically, we carefully design a vibration network branch and a Fourier fusion module, to detect and mitigate vibration noises. The fusion framework is compatible with popular vision-only algorithms. Extensive validation on the multimodal dataset demonstrates superior performance and robustness against vision-only algorithms. Without the need for large external equipment, our V$^2$-SfMLearner has the potential for integration into clinical capsule robots, providing real-time and dependable digestive examination tools. The findings show promise for practical implementation in clinical settings, enhancing the diagnostic capabilities of doctors.