Quadrant Segmentation VLM with Few-Shot Adaptation and OCT Learning-based Explainability Methods for Diabetic Retinopathy

TL;DR

This paper introduces a multimodal explainability model for diabetic retinopathy that analyzes lesion distributions in retinal quadrants using few-shot learning and OCT-based methods, enhancing interpretability and diagnostic accuracy.

Contribution

The paper proposes a novel VLM-based multimodal explainability approach that mimics ophthalmologist reasoning by analyzing lesion distributions across retinal quadrants with few-shot learning.

Findings

Effective identification of DR lesions in fundus and OCT images.

Generation of Grad-CAM heatmaps highlighting regions contributing to diagnosis.

Addresses limitations of current single-modality, one-dimensional models.

Abstract

Diabetic Retinopathy (DR) is a leading cause of vision loss worldwide, requiring early detection to preserve sight. Limited access to physicians often leaves DR undiagnosed. To address this, AI models utilize lesion segmentation for interpretability; however, manually annotating lesions is impractical for clinicians. Physicians require a model that explains the reasoning for classifications rather than just highlighting lesion locations. Furthermore, current models are one-dimensional, relying on a single imaging modality for explainability and achieving limited effectiveness. In contrast, a quantitative-detection system that identifies individual DR lesions in natural language would overcome these limitations, enabling diverse applications in screening, treatment, and research settings. To address this issue, this paper presents a novel multimodal explainability model utilizing a VLM with few-shot learning, which mimics an ophthalmologist's reasoning by analyzing lesion distributions within retinal quadrants for fundus images. The model generates paired Grad-CAM heatmaps, showcasing individual neuron weights across both OCT and fundus images, which visually highlight the regions contributing to DR severity classification. Using a dataset of 3,000 fundus images and 1,000 OCT images, this innovative methodology addresses key limitations in current DR diagnostics, offering a practical and comprehensive tool for improving patient outcomes.