CheXpedition: Investigating Generalization Challenges for Translation of Chest X-Ray Algorithms to the Clinical Setting

TL;DR

This study evaluates the generalization of top chest X-ray deep learning models across different tasks, datasets, and settings, highlighting their potential and challenges for clinical deployment.

Contribution

It systematically assesses the performance of leading models on diverse tasks and datasets, revealing their strengths and limitations for clinical translation.

Findings

Top models achieve high AUC on TB detection without fine-tuning.

Models perform similarly on photos and original X-ray images.

External dataset performance is comparable or better than radiologists.

Abstract

Although there have been several recent advances in the application of deep learning algorithms to chest x-ray interpretation, we identify three major challenges for the translation of chest x-ray algorithms to the clinical setting. We examine the performance of the top 10 performing models on the CheXpert challenge leaderboard on three tasks: (1) TB detection, (2) pathology detection on photos of chest x-rays, and (3) pathology detection on data from an external institution. First, we find that the top 10 chest x-ray models on the CheXpert competition achieve an average AUC of 0.851 on the task of detecting TB on two public TB datasets without fine-tuning or including the TB labels in training data. Second, we find that the average performance of the models on photos of x-rays (AUC = 0.916) is similar to their performance on the original chest x-ray images (AUC = 0.924). Third, we find that the models tested on an external dataset either perform comparably to or exceed the average performance of radiologists. We believe that our investigation will inform rapid translation of deep learning algorithms to safe and effective clinical decision support tools that can be validated prospectively with large impact studies and clinical trials.