XDR-LVLM: An Explainable Vision-Language Large Model for Diabetic Retinopathy Diagnosis

TL;DR

XDR-LVLM is an explainable vision-language model that improves diabetic retinopathy diagnosis accuracy and provides natural language explanations, enhancing clinical interpretability and trust.

Contribution

This paper introduces XDR-LVLM, a novel framework combining vision-language models with multi-task prompt engineering for accurate, interpretable diabetic retinopathy diagnosis.

Findings

Achieves state-of-the-art accuracy with 84.55% balanced accuracy

Generates comprehensive diagnostic reports with high fluency and clinical relevance

Outperforms previous methods in concept detection and diagnosis metrics

Abstract

Diabetic Retinopathy (DR) is a major cause of global blindness, necessitating early and accurate diagnosis. While deep learning models have shown promise in DR detection, their black-box nature often hinders clinical adoption due to a lack of transparency and interpretability. To address this, we propose XDR-LVLM (eXplainable Diabetic Retinopathy Diagnosis with LVLM), a novel framework that leverages Vision-Language Large Models (LVLMs) for high-precision DR diagnosis coupled with natural language-based explanations. XDR-LVLM integrates a specialized Medical Vision Encoder, an LVLM Core, and employs Multi-task Prompt Engineering and Multi-stage Fine-tuning to deeply understand pathological features within fundus images and generate comprehensive diagnostic reports. These reports explicitly include DR severity grading, identification of key pathological concepts (e.g., hemorrhages, exudates, microaneurysms), and detailed explanations linking observed features to the diagnosis. Extensive experiments on the Diabetic Retinopathy (DDR) dataset demonstrate that XDR-LVLM achieves state-of-the-art performance, with a Balanced Accuracy of 84.55% and an F1 Score of 79.92% for disease diagnosis, and superior results for concept detection (77.95% BACC, 66.88% F1). Furthermore, human evaluations confirm the high fluency, accuracy, and clinical utility of the generated explanations, showcasing XDR-LVLM's ability to bridge the gap between automated diagnosis and clinical needs by providing robust and interpretable insights.