M4CXR: Exploring Multi-task Potentials of Multi-modal Large Language Models for Chest X-ray Interpretation

TL;DR

M4CXR is a multi-modal large language model that enhances chest X-ray interpretation by supporting multiple tasks like report generation, visual grounding, and VQA with state-of-the-art clinical accuracy.

Contribution

The paper introduces M4CXR, a multi-task, multi-modal LLM trained on a visual instruction dataset, achieving superior performance across several CXR analysis tasks.

Findings

M4CXR achieves state-of-the-art accuracy in medical report generation.

The model performs comparably to specialized models in visual grounding.

M4CXR demonstrates outstanding performance in visual question answering.

Abstract

The rapid evolution of artificial intelligence, especially in large language models (LLMs), has significantly impacted various domains, including healthcare. In chest X-ray (CXR) analysis, previous studies have employed LLMs, but with limitations: either underutilizing the multi-tasking capabilities of LLMs or lacking clinical accuracy. This paper presents M4CXR, a multi-modal LLM designed to enhance CXR interpretation. The model is trained on a visual instruction-following dataset that integrates various task-specific datasets in a conversational format. As a result, the model supports multiple tasks such as medical report generation (MRG), visual grounding, and visual question answering (VQA). M4CXR achieves state-of-the-art clinical accuracy in MRG by employing a chain-of-thought prompting strategy, in which it identifies findings in CXR images and subsequently generates corresponding reports. The model is adaptable to various MRG scenarios depending on the available inputs, such as single-image, multi-image, and multi-study contexts. In addition to MRG, M4CXR performs visual grounding at a level comparable to specialized models and also demonstrates outstanding performance in VQA. Both quantitative and qualitative assessments reveal M4CXR's versatility in MRG, visual grounding, and VQA, while consistently maintaining clinical accuracy.