LMOD: A Large Multimodal Ophthalmology Dataset and Benchmark for Large Vision-Language Models

TL;DR

This paper introduces LMOD, a comprehensive ophthalmology dataset and benchmark for evaluating large vision-language models, revealing significant performance gaps and failure modes in current models compared to supervised methods.

Contribution

The paper presents LMOD, a large-scale multimodal ophthalmology benchmark, and evaluates 13 LVLMs, highlighting their limitations and the need for specialized ophthalmology models.

Findings

LVLMs perform significantly worse in ophthalmology tasks.

Six major failure modes identified in LVLMs.

Supervised models outperform LVLMs in accuracy.

Abstract

The prevalence of vision-threatening eye diseases is a significant global burden, with many cases remaining undiagnosed or diagnosed too late for effective treatment. Large vision-language models (LVLMs) have the potential to assist in understanding anatomical information, diagnosing eye diseases, and drafting interpretations and follow-up plans, thereby reducing the burden on clinicians and improving access to eye care. However, limited benchmarks are available to assess LVLMs' performance in ophthalmology-specific applications. In this study, we introduce LMOD, a large-scale multimodal ophthalmology benchmark consisting of 21,993 instances across (1) five ophthalmic imaging modalities: optical coherence tomography, color fundus photographs, scanning laser ophthalmoscopy, lens photographs, and surgical scenes; (2) free-text, demographic, and disease biomarker information; and (3) primary ophthalmology-specific applications such as anatomical information understanding, disease diagnosis, and subgroup analysis. In addition, we benchmarked 13 state-of-the-art LVLM representatives from closed-source, open-source, and medical domains. The results demonstrate a significant performance drop for LVLMs in ophthalmology compared to other domains. Systematic error analysis further identified six major failure modes: misclassification, failure to abstain, inconsistent reasoning, hallucination, assertions without justification, and lack of domain-specific knowledge. In contrast, supervised neural networks specifically trained on these tasks as baselines demonstrated high accuracy. These findings underscore the pressing need for benchmarks in the development and validation of ophthalmology-specific LVLMs.