RadGPT: Constructing 3D Image-Text Tumor Datasets

TL;DR

This paper introduces AbdomenAtlas 3.0, a comprehensive abdominal CT dataset with expert-reviewed reports and tumor masks, and presents RadGPT, a novel framework for automated report generation that enhances tumor detection accuracy.

Contribution

The paper provides the first high-quality, publicly available abdominal CT dataset with detailed reports and introduces RadGPT, a new method for automated, segmentation-assisted radiology report generation.

Findings

RadGPT effectively converts tumor masks into structured reports.

Segmentation significantly improves AI tumor detection in reports.

AbdomenAtlas 3.0 expands tumor data and standardizes radiology reporting.

Abstract

Cancers identified in CT scans are usually accompanied by detailed radiology reports, but publicly available CT datasets often lack these essential reports. This absence limits their usefulness for developing accurate report generation AI. To address this gap, we present AbdomenAtlas 3.0, the first public, high-quality abdominal CT dataset with detailed, expert-reviewed radiology reports. All reports are paired with per-voxel masks and they describe liver, kidney and pancreatic tumors. AbdomenAtlas 3.0 has 9,262 triplets of CT, mask and report--3,955 with tumors. These CT scans come from 17 public datasets. Besides creating the reports for these datasets, we expanded their number of tumor masks by 4.2x, identifying 3,011 new tumor cases. Notably, the reports in AbdomenAtlas 3.0 are more standardized, and generated faster than traditional human-made reports. They provide details like tumor size, location, attenuation and surgical resectability. These reports were created by 12 board-certified radiologists using our proposed RadGPT, a novel framework that converted radiologist-revised tumor segmentation masks into structured and narrative reports. Besides being a dataset creation tool, RadGPT can also become a fully-automatic, segmentation-assisted report generation method. We benchmarked this method and 5 state-of-the-art report generation vision-language models. Our results show that segmentation strongly improves tumor detection in AI-made reports.