MMXU: A Multi-Modal and Multi-X-ray Understanding Dataset for Disease Progression

TL;DR

This paper introduces MMXU, a new dataset for medical visual question answering that emphasizes disease progression analysis over time, and proposes a method to improve LVLMs by incorporating historical patient records.

Contribution

The paper presents MMXU, a novel multi-image dataset for MedVQA focusing on disease progression, and introduces MAG, a method that integrates historical records to enhance diagnostic accuracy.

Findings

Incorporating historical records improves diagnostic accuracy by at least 20%.

Current LVLMs struggle with disease progression tasks on MMXU.

Fine-tuning with MAG enhances model performance significantly.

Abstract

Large vision-language models (LVLMs) have shown great promise in medical applications, particularly in visual question answering (MedVQA) and diagnosis from medical images. However, existing datasets and models often fail to consider critical aspects of medical diagnostics, such as the integration of historical records and the analysis of disease progression over time. In this paper, we introduce MMXU (Multimodal and MultiX-ray Understanding), a novel dataset for MedVQA that focuses on identifying changes in specific regions between two patient visits. Unlike previous datasets that primarily address single-image questions, MMXU enables multi-image questions, incorporating both current and historical patient data. We demonstrate the limitations of current LVLMs in identifying disease progression on MMXU-\textit{test}, even those that perform well on traditional benchmarks. To address this, we propose a MedRecord-Augmented Generation (MAG) approach, incorporating both global and regional historical records. Our experiments show that integrating historical records significantly enhances diagnostic accuracy by at least 20\%, bridging the gap between current LVLMs and human expert performance. Additionally, we fine-tune models with MAG on MMXU-\textit{dev}, which demonstrates notable improvements. We hope this work could illuminate the avenue of advancing the use of LVLMs in medical diagnostics by emphasizing the importance of historical context in interpreting medical images. Our dataset is released at github: https://github.com/linjiemu/MMXU.