AnatomiX, an Anatomy-Aware Grounded Multimodal Large Language Model for Chest X-Ray Interpretation

TL;DR

AnatomiX is a novel multimodal large language model that enhances anatomical grounding and reasoning in chest X-ray interpretation by combining structured anatomical understanding with downstream medical tasks, outperforming existing methods.

Contribution

The paper introduces AnatomiX, a two-stage anatomically grounded multimodal model that significantly improves spatial reasoning and anatomical understanding in chest X-ray analysis.

Findings

Over 25% improvement in anatomy grounding accuracy

Superior performance in phrase grounding and diagnosis tasks

Effective integration of anatomical features with language models

Abstract

Multimodal medical large language models have shown substantial progress in chest X-ray interpretation but continue to face challenges in spatial reasoning and anatomical understanding. Although existing grounding techniques improve overall performance, they often fail to establish a true anatomical correspondence, resulting in incorrect anatomical understanding in the medical domain. To address this gap, we introduce AnatomiX, a multitask multimodal large language model for anatomically grounded chest X-ray interpretation. Inspired by the radiological workflow, AnatomiX adopts a two stage approach: first, it identifies anatomical structures and extracts their features, and then leverages a large language model to perform diverse downstream tasks such as phrase grounding, report generation, visual question answering, and image understanding. Extensive experiments across multiple benchmarks demonstrate that AnatomiX achieves superior anatomical reasoning and delivers over 25% improvement in performance on anatomy grounding, phrase grounding, grounded diagnosis and grounded captioning tasks compared to existing approaches. Code and pretrained model are available at github.com/aneesurhashmi/anatomix.