RadImageNet-VQA: A Large-Scale CT and MRI Dataset for Radiologic Visual Question Answering

TL;DR

RadImageNet-VQA introduces a comprehensive large-scale dataset for radiologic VQA, enabling advancements in medical image understanding and diagnosis through diverse tasks and extensive annotations.

Contribution

This work provides the first large-scale, expert-annotated CT and MRI dataset for radiologic VQA, covering multiple tasks and ensuring robustness against text-based shortcuts.

Findings

State-of-the-art models struggle with fine-grained pathology identification.

Model performance drops to near-random without image inputs.

Dataset is free from linguistic shortcuts and publicly available.

Abstract

In this work, we introduce RadImageNet-VQA, a large-scale dataset designed to advance radiologic visual question answering (VQA) on CT and MRI exams. Existing medical VQA datasets are limited in scale, dominated by X-ray imaging or biomedical illustrations, and often prone to text-based shortcuts. RadImageNet-VQA is built from expert-curated annotations and provides 750K images paired with 7.5M question-answer samples. It covers three key tasks - abnormality detection, anatomy recognition, and pathology identification - spanning eight anatomical regions and 97 pathology categories, and supports open-ended, closed-ended, and multiple-choice questions. Extensive experiments show that state-of-the-art vision-language models still struggle with fine-grained pathology identification, particularly in open-ended settings and even after fine-tuning. Text-only analysis further reveals that model performance collapses to near-random without image inputs, confirming that RadImageNet-VQA is free from linguistic shortcuts. The full dataset and benchmark are publicly available at https://huggingface.co/datasets/raidium/RadImageNet-VQA.