Benchmarking Chest X-ray Diagnosis Models Across Multinational Datasets

TL;DR

This study benchmarks foundation models and CNNs on multinational chest X-ray datasets, demonstrating that foundation models generally outperform CNNs, especially with knowledge-enhanced prompts, but all models perform worse on pediatric cases.

Contribution

It provides a comprehensive evaluation of foundation models versus CNNs across diverse datasets and tasks, highlighting the impact of prompt design and structured supervision.

Findings

Foundation models outperform CNNs in accuracy and task coverage.

MAVL achieved the highest performance among evaluated models.

All models show reduced accuracy on pediatric chest X-ray cases.

Abstract

Foundation models leveraging vision-language pretraining have shown promise in chest X-ray (CXR) interpretation, yet their real-world performance across diverse populations and diagnostic tasks remains insufficiently evaluated. This study benchmarks the diagnostic performance and generalizability of foundation models versus traditional convolutional neural networks (CNNs) on multinational CXR datasets. We evaluated eight CXR diagnostic models - five vision-language foundation models and three CNN-based architectures - across 37 standardized classification tasks using six public datasets from the USA, Spain, India, and Vietnam, and three private datasets from hospitals in China. Performance was assessed using AUROC, AUPRC, and other metrics across both shared and dataset-specific tasks. Foundation models outperformed CNNs in both accuracy and task coverage. MAVL, a model incorporating knowledge-enhanced prompts and structured supervision, achieved the highest performance on public (mean AUROC: 0.82; AUPRC: 0.32) and private (mean AUROC: 0.95; AUPRC: 0.89) datasets, ranking first in 14 of 37 public and 3 of 4 private tasks. All models showed reduced performance on pediatric cases, with average AUROC dropping from 0.88 +/- 0.18 in adults to 0.57 +/- 0.29 in children (p = 0.0202). These findings highlight the value of structured supervision and prompt design in radiologic AI and suggest future directions including geographic expansion and ensemble modeling for clinical deployment. Code for all evaluated models is available at https://drive.google.com/drive/folders/1B99yMQm7bB4h1sVMIBja0RfUu8gLktCE