RadGenome-Chest CT: A Grounded Vision-Language Dataset for Chest CT Analysis

TL;DR

RadGenome-Chest CT is a large-scale, region-guided 3D chest CT dataset with detailed annotations, reports, and VQA pairs, designed to advance multimodal foundation models in medical imaging.

Contribution

The paper introduces RadGenome-Chest CT, a comprehensive dataset with organ segmentation, grounded reports, and VQA pairs, enhancing multimodal AI research in chest CT analysis.

Findings

Dataset includes 25,692 CT volumes and reports from 20,000 patients.

Contains 197 organ segmentation categories and 665K grounded reports.

Provides 1.3 million grounded VQA pairs for multimodal learning.

Abstract

Developing generalist foundation model has recently attracted tremendous attention among researchers in the field of AI for Medicine (AI4Medicine). A pivotal insight in developing these models is their reliance on dataset scaling, which emphasizes the requirements on developing open-source medical image datasets that incorporate diverse supervision signals across various imaging modalities. In this paper, we introduce RadGenome-Chest CT, a comprehensive, large-scale, region-guided 3D chest CT interpretation dataset based on CT-RATE. Specifically, we leverage the latest powerful universal segmentation and large language models, to extend the original datasets (over 25,692 non-contrast 3D chest CT volume and reports from 20,000 patients) from the following aspects: (i) organ-level segmentation masks covering 197 categories, which provide intermediate reasoning visual clues for interpretation; (ii) 665 K multi-granularity grounded reports, where each sentence of the report is linked to the corresponding anatomical region of CT volume in the form of a segmentation mask; (iii) 1.3 M grounded VQA pairs, where questions and answers are all linked with reference segmentation masks, enabling models to associate visual evidence with textual explanations. All grounded reports and VQA pairs in the validation set have gone through manual verification to ensure dataset quality. We believe that RadGenome-Chest CT can significantly advance the development of multimodal medical foundation models, by training to generate texts based on given segmentation regions, which is unattainable with previous relevant datasets. We will release all segmentation masks, grounded reports, and VQA pairs to facilitate further research and development in this field.