VisualCheXbert: Addressing the Discrepancy Between Radiology Report Labels and Image Labels

TL;DR

This paper introduces VisualCheXbert, a model that improves the alignment of radiology report labels with actual image labels, addressing discrepancies caused by radiologist labeling inconsistencies, to enhance medical image interpretation.

Contribution

We propose VisualCheXbert, a novel method using a biomedically-pretrained BERT to better align report-derived labels with image labels, outperforming existing report labelers.

Findings

VisualCheXbert achieves higher F1 scores than existing labelers.

It better aligns with radiologists' image labels than report labels.

Improves supervision quality for medical image analysis models.

Abstract

Automatic extraction of medical conditions from free-text radiology reports is critical for supervising computer vision models to interpret medical images. In this work, we show that radiologists labeling reports significantly disagree with radiologists labeling corresponding chest X-ray images, which reduces the quality of report labels as proxies for image labels. We develop and evaluate methods to produce labels from radiology reports that have better agreement with radiologists labeling images. Our best performing method, called VisualCheXbert, uses a biomedically-pretrained BERT model to directly map from a radiology report to the image labels, with a supervisory signal determined by a computer vision model trained to detect medical conditions from chest X-ray images. We find that VisualCheXbert outperforms an approach using an existing radiology report labeler by an average F1 score of 0.14 (95% CI 0.12, 0.17). We also find that VisualCheXbert better agrees with radiologists labeling chest X-ray images than do radiologists labeling the corresponding radiology reports by an average F1 score across several medical conditions of between 0.12 (95% CI 0.09, 0.15) and 0.21 (95% CI 0.18, 0.24).