When are radiology reports useful for training medical image classifiers?

TL;DR

This paper systematically studies how radiology reports can be used during pre-training and fine-tuning to improve medical image classification, revealing when report-based information is most beneficial.

Contribution

It provides a comprehensive analysis of report utilization during training, highlighting conditions where report-based pre-training and fine-tuning improve classification performance.

Findings

Report-based pre-training helps when labels are well-represented in text.

Explicit image-text alignment can be harmful if labels are weakly associated.

Fine-tuning with reports can outperform pre-training in certain scenarios.

Abstract

Medical images used to train machine learning models are often accompanied by radiology reports containing rich expert annotations. However, relying on these reports as inputs for clinical prediction requires the timely manual work of a trained radiologist. This raises a natural question: when can radiology reports be leveraged during training to improve image-only classification? Prior works are limited to evaluating pre-trained image representations by fine-tuning them to predict diagnostic labels, often extracted from reports, ignoring tasks with labels that are weakly associated with the text. To address this gap, we conduct a systematic study of how radiology reports can be used during both pre-training and fine-tuning, across diagnostic and prognostic tasks (e.g., 12-month readmission), and under varying training set sizes. Our findings reveal that: (1) Leveraging reports during pre-training is beneficial for downstream classification tasks where the label is well-represented in the text; however, pre-training through explicit image-text alignment can be detrimental in settings where it's not; (2) Fine-tuning with reports can lead to significant improvements and even have a larger impact than the pre-training method in certain settings. These results provide actionable insights into when and how to leverage privileged text data to train medical image classifiers while highlighting gaps in current research.