Learning Segmentation from Radiology Reports

TL;DR

This paper introduces R-Super, a novel report-supervision loss that uses radiology reports as voxel-wise supervision to significantly enhance tumor segmentation accuracy in CT scans, especially when segmentation masks are limited.

Contribution

The paper presents a new method to leverage radiology reports for improving tumor segmentation, addressing data scarcity issues in medical imaging AI.

Findings

F1 Score improved by up to 16% with R-Super.

Effective even with very few training masks (e.g., 50).

Significant performance gains when many masks are available.

Abstract

Tumor segmentation in CT scans is key for diagnosis, surgery, and prognosis, yet segmentation masks are scarce because their creation requires time and expertise. Public abdominal CT datasets have from dozens to a couple thousand tumor masks, but hospitals have hundreds of thousands of tumor CTs with radiology reports. Thus, leveraging reports to improve segmentation is key for scaling. In this paper, we propose a report-supervision loss (R-Super) that converts radiology reports into voxel-wise supervision for tumor segmentation AI. We created a dataset with 6,718 CT-Report pairs (from the UCSF Hospital), and merged it with public CT-Mask datasets (from AbdomenAtlas 2.0). We used our R-Super to train with these masks and reports, and strongly improved tumor segmentation in internal and external validation--F1 Score increased by up to 16% with respect to training with masks only. By leveraging readily available radiology reports to supplement scarce segmentation masks, R-Super strongly improves AI performance both when very few training masks are available (e.g., 50), and when many masks were available (e.g., 1.7K). Project: https://github.com/MrGiovanni/R-Super