Generalizing monocular colonoscopy image depth estimation by uncertainty-based global and local fusion network

TL;DR

This paper introduces a novel CNN-Transformer framework with an uncertainty-based fusion block that significantly improves depth estimation and generalization in colonoscopy images, enabling robust clinical applications without fine-tuning.

Contribution

The study presents a new deep learning architecture combining local and global features with uncertainty-based fusion, enhancing depth estimation generalization in endoscopic images.

Findings

Excellent generalization across multiple datasets

Robust depth estimation in real clinical scenarios

No fine-tuning required for unseen data

Abstract

Objective: Depth estimation is crucial for endoscopic navigation and manipulation, but obtaining ground-truth depth maps in real clinical scenarios, such as the colon, is challenging. This study aims to develop a robust framework that generalizes well to real colonoscopy images, overcoming challenges like non-Lambertian surface reflection and diverse data distributions. Methods: We propose a framework combining a convolutional neural network (CNN) for capturing local features and a Transformer for capturing global information. An uncertainty-based fusion block was designed to enhance generalization by identifying complementary contributions from the CNN and Transformer branches. The network can be trained with simulated datasets and generalize directly to unseen clinical data without any fine-tuning. Results: Our method is validated on multiple datasets and demonstrates an excellent generalization ability across various datasets and anatomical structures. Furthermore, qualitative analysis in real clinical scenarios confirmed the robustness of the proposed method. Conclusion: The integration of local and global features through the CNN-Transformer architecture, along with the uncertainty-based fusion block, improves depth estimation performance and generalization in both simulated and real-world endoscopic environments. Significance: This study offers a novel approach to estimate depth maps for endoscopy images despite the complex conditions in clinic, serving as a foundation for endoscopic automatic navigation and other clinical tasks, such as polyp detection and segmentation.