Detecting Shortcuts in Medical Images -- A Case Study in Chest X-rays

TL;DR

This paper highlights the issue of shortcuts and artifacts in medical image datasets, demonstrating their impact on model performance and emphasizing the need for careful data validation and subgroup testing in chest X-ray classification.

Contribution

The study validates previous concerns about shortcuts in medical imaging datasets and provides a detailed case study with annotations and recommendations for improving model robustness.

Findings

Models may exploit shortcuts, leading to overestimated performance.

Annotated subset of pneumothorax images with drains to improve data quality.

Recommendations for better dataset validation and subgroup testing.

Abstract

The availability of large public datasets and the increased amount of computing power have shifted the interest of the medical community to high-performance algorithms. However, little attention is paid to the quality of the data and their annotations. High performance on benchmark datasets may be reported without considering possible shortcuts or artifacts in the data, besides, models are not tested on subpopulation groups. With this work, we aim to raise awareness about shortcuts problems. We validate previous findings, and present a case study on chest X-rays using two publicly available datasets. We share annotations for a subset of pneumothorax images with drains. We conclude with general recommendations for medical image classification.