Validation of a deep learning mammography model in a population with low screening rates

TL;DR

This study evaluates a deep learning mammography model trained on US and UK data, demonstrating its effective generalization to a Chinese hospital setting with low screening rates, and proposes a variance-based filtering method for deployment.

Contribution

It provides empirical evidence that a model trained on advantaged populations can generalize well to under-served populations and introduces a simple filtering technique for clinical deployment.

Findings

Model performs similarly in new population despite different institution and demographics.

Variance-based filtering improves prediction reliability in new settings.

Deep learning can aid in increasing access to mammography screening in low-rate populations.

Abstract

A key promise of AI applications in healthcare is in increasing access to quality medical care in under-served populations and emerging markets. However, deep learning models are often only trained on data from advantaged populations that have the infrastructure and resources required for large-scale data collection. In this paper, we aim to empirically investigate the potential impact of such biases on breast cancer detection in mammograms. We specifically explore how a deep learning algorithm trained on screening mammograms from the US and UK generalizes to mammograms collected at a hospital in China, where screening is not widely implemented. For the evaluation, we use a top-scoring model developed for the Digital Mammography DREAM Challenge. Despite the change in institution and population composition, we find that the model generalizes well, exhibiting similar performance to that achieved in the DREAM Challenge, even when controlling for tumor size. We also illustrate a simple but effective method for filtering predictions based on model variance, which can be particularly useful for deployment in new settings. While there are many components in developing a clinically effective system, these results represent a promising step towards increasing access to life-saving screening mammography in populations where screening rates are currently low.