Endo-CLIP: Progressive Self-Supervised Pre-training on Raw Colonoscopy Records

TL;DR

Endo-CLIP is a self-supervised pre-training framework tailored for colonoscopy image-text data, improving polyp detection and classification by addressing background noise, complex terminology, and multi-lesion ambiguity.

Contribution

It introduces a three-stage framework that cleans data, uses language models for attribute extraction, and applies cross-attention to enhance contrastive learning in medical imaging.

Findings

Outperforms existing methods in zero-shot polyp detection

Achieves higher accuracy in few-shot classification

Demonstrates robustness across diverse colonoscopy datasets

Abstract

Pre-training on image-text colonoscopy records offers substantial potential for improving endoscopic image analysis, but faces challenges including non-informative background images, complex medical terminology, and ambiguous multi-lesion descriptions. We introduce Endo-CLIP, a novel self-supervised framework that enhances Contrastive Language-Image Pre-training (CLIP) for this domain. Endo-CLIP's three-stage framework--cleansing, attunement, and unification--addresses these challenges by (1) removing background frames, (2) leveraging large language models to extract clinical attributes for fine-grained contrastive learning, and (3) employing patient-level cross-attention to resolve multi-polyp ambiguities. Extensive experiments demonstrate that Endo-CLIP significantly outperforms state-of-the-art pre-training methods in zero-shot and few-shot polyp detection and classification, paving the way for more accurate and clinically relevant endoscopic analysis.