RadTex: Learning Efficient Radiograph Representations from Text Reports

TL;DR

RadTex introduces a data-efficient framework that leverages radiology reports and image-captioning pretraining to enhance chest radiograph classification with limited labeled data, outperforming traditional pretraining methods.

Contribution

The paper proposes a novel radiology report-based pretraining approach that improves medical image classification performance in low-data scenarios.

Findings

Outperforms ImageNet-supervised pretraining on limited data

Achieves higher accuracy across 9 pathologies

Effective with fewer than 1000 labeled examples

Abstract

Automated analysis of chest radiography using deep learning has tremendous potential to enhance the clinical diagnosis of diseases in patients. However, deep learning models typically require large amounts of annotated data to achieve high performance -- often an obstacle to medical domain adaptation. In this paper, we build a data-efficient learning framework that utilizes radiology reports to improve medical image classification performance with limited labeled data (fewer than 1000 examples). Specifically, we examine image-captioning pretraining to learn high-quality medical image representations that train on fewer examples. Following joint pretraining of a convolutional encoder and transformer decoder, we transfer the learned encoder to various classification tasks. Averaged over 9 pathologies, we find that our model achieves higher classification performance than ImageNet-supervised and in-domain supervised pretraining when labeled training data is limited.