Generalized Radiograph Representation Learning via Cross-supervision between Images and Free-text Radiology Reports

TL;DR

This paper introduces REFERS, a cross-supervised learning method that leverages free-text radiology reports to pre-train vision transformers, outperforming traditional supervised and self-supervised approaches in radiograph analysis.

Contribution

REFERS is a novel cross-supervised pre-training approach that uses radiology reports as supervision signals, reducing reliance on labor-intensive annotations and surpassing existing methods.

Findings

Outperforms transfer and self-supervised methods on 4 X-ray datasets

Surpasses methods with structured labels and source domain supervision

Effective with extremely limited supervision

Abstract

Pre-training lays the foundation for recent successes in radiograph analysis supported by deep learning. It learns transferable image representations by conducting large-scale fully-supervised or self-supervised learning on a source domain. However, supervised pre-training requires a complex and labor intensive two-stage human-assisted annotation process while self-supervised learning cannot compete with the supervised paradigm. To tackle these issues, we propose a cross-supervised methodology named REviewing FreE-text Reports for Supervision (REFERS), which acquires free supervision signals from original radiology reports accompanying the radiographs. The proposed approach employs a vision transformer and is designed to learn joint representations from multiple views within every patient study. REFERS outperforms its transfer learning and self-supervised learning counterparts on 4 well-known X-ray datasets under extremely limited supervision. Moreover, REFERS even surpasses methods based on a source domain of radiographs with human-assisted structured labels. Thus REFERS has the potential to replace canonical pre-training methodologies.