A Generative Framework for Bidirectional Image-Report Understanding in Chest Radiography

TL;DR

This paper introduces MAViLT, a novel framework that significantly improves multimodal reasoning and report generation in chest X-ray analysis using large language models, addressing key challenges in medical vision-language tasks.

Contribution

The paper presents MAViLT, a new multi-stage adaptive tuning framework that enhances LLMs for accurate medical image-report understanding and generation in chest radiography.

Findings

Achieves state-of-the-art results on MIMIC-CXR and Indiana datasets.

Enables accurate report generation and image synthesis from text.

Validated by human evaluations for clinical relevance.

Abstract

The rapid advancements in large language models (LLMs) have unlocked their potential for multimodal tasks, where text and visual data are processed jointly. However, applying LLMs to medical imaging, particularly for chest X-rays (CXR), poses significant challenges due to the need for precise visual-textual alignment and the preservation of critical diagnostic details. In this paper, we propose Multi-Stage Adaptive Vision-Language Tuning (MAViLT), a novel framework designed to enhance multimodal reasoning and generation for CXR understanding. MAViLT incorporates a clinical gradient-weighted tokenization process and a hierarchical fine-tuning strategy, enabling it to generate accurate radiology reports, synthesize realistic CXRs from text, and answer vision-based clinical questions. We evaluate MAViLT on two benchmark datasets, MIMIC-CXR and Indiana University CXR, achieving state-of-the-art results across all tasks. Human evaluations further validate the clinical relevance and utility of MAViLT, making it a robust tool for real-world medical applications. This work demonstrates the feasibility of leveraging LLMs for multimodal medical imaging while addressing key challenges in vision-language integration.