VIGIL: Vision-Language Guided Multiple Instance Learning Framework for Ulcerative Colitis Histological Healing Prediction

TL;DR

VIGIL is a novel vision-language guided multiple instance learning framework that improves ulcerative colitis histological healing prediction by integrating endoscopic images and diagnostic reports, reducing annotation needs and enhancing accuracy.

Contribution

VIGIL introduces a dual-branch MIL model with image-text alignment and multi-modal fusion, pioneering the combination of vision and language guidance in UC histological prediction.

Findings

Achieves 92.69% accuracy and 94.79% AUC on clinical dataset

Outperforms existing state-of-the-art methods

Reduces annotation burden while improving prediction reliability

Abstract

Objective: Ulcerative colitis (UC), characterized by chronic inflammation with alternating remission-relapse cycles, requires precise histological healing (HH) evaluation to improve clinical outcomes. To overcome the limitations of annotation-intensive deep learning methods and suboptimal multi-instance learning (MIL) in HH prediction, we propose VIGIL, the first vision-language guided MIL framework integrating white light endoscopy (WLE) and endocytoscopy (EC). Methods:VIGIL begins with a dual-branch MIL module KS-MIL based on top-K typical frames selection and similarity metric adaptive learning to learn relationships among frame features effectively. By integrating the diagnostic report text and specially designed multi-level alignment and supervision between image-text pairs, VIGIL establishes joint image-text guidance during training to capture richer disease-related semantic information. Furthermore, VIGIL employs a multi-modal masked relation fusion (MMRF) strategy to uncover the latent diagnostic correlations of two endoscopic image representations. Results:Comprehensive experiments on a real-world clinical dataset demonstrate VIGIL's superior performance, achieving 92.69\% accuracy and 94.79\% AUC, outperforming existing state-of-the-art methods. Conclusion: The proposed VIGIL framework successfully establishes an effective vision-language guided MIL paradigm for UC HH prediction, reducing annotation burdens while improving prediction reliability. Significance: The research outcomes provide new insights for non-invasive UC diagnosis and hold theoretical significance and clinical value for advancing intelligent healthcare development.