EndoFinder: Online Image Retrieval for Explainable Colorectal Polyp Diagnosis

TL;DR

EndoFinder is an explainable image retrieval system for colorectal polyps that aids diagnosis by finding similar reference images, combining self-supervised learning with clinical interpretability during colonoscopy.

Contribution

It introduces a novel polyp-aware image encoder trained with self-supervised learning, enabling effective image retrieval for explainable diagnosis during colonoscopy.

Findings

Achieves comparable performance to supervised models in polyp re-identification

Provides explainable diagnostics through image retrieval

Supports real-time decision-making during colonoscopy procedures

Abstract

Determining the necessity of resecting malignant polyps during colonoscopy screen is crucial for patient outcomes, yet challenging due to the time-consuming and costly nature of histopathology examination. While deep learning-based classification models have shown promise in achieving optical biopsy with endoscopic images, they often suffer from a lack of explainability. To overcome this limitation, we introduce EndoFinder, a content-based image retrieval framework to find the 'digital twin' polyp in the reference database given a newly detected polyp. The clinical semantics of the new polyp can be inferred referring to the matched ones. EndoFinder pioneers a polyp-aware image encoder that is pre-trained on a large polyp dataset in a self-supervised way, merging masked image modeling with contrastive learning. This results in a generic embedding space ready for different downstream clinical tasks based on image retrieval. We validate the framework on polyp re-identification and optical biopsy tasks, with extensive experiments demonstrating that EndoFinder not only achieves explainable diagnostics but also matches the performance of supervised classification models. EndoFinder's reliance on image retrieval has the potential to support diverse downstream decision-making tasks during real-time colonoscopy procedures.