Fair Evaluation of Federated Learning Algorithms for Automated Breast Density Classification: The Results of the 2022 ACR-NCI-NVIDIA Federated Learning Challenge

TL;DR

This paper reports on the 2022 federated learning challenge for breast density classification, demonstrating that FL can achieve performance comparable to centralized models while preserving data privacy across multiple medical facilities.

Contribution

It presents a comprehensive evaluation of federated learning methods for breast density classification through a global challenge, highlighting effective strategies and performance benchmarks.

Findings

Winning model achieved a linear kappa of 0.653 on challenge data

External dataset kappa score was 0.413, showing good generalization

FL models performed comparably to centralized models

Abstract

The correct interpretation of breast density is important in the assessment of breast cancer risk. AI has been shown capable of accurately predicting breast density, however, due to the differences in imaging characteristics across mammography systems, models built using data from one system do not generalize well to other systems. Though federated learning (FL) has emerged as a way to improve the generalizability of AI without the need to share data, the best way to preserve features from all training data during FL is an active area of research. To explore FL methodology, the breast density classification FL challenge was hosted in partnership with the American College of Radiology, Harvard Medical School's Mass General Brigham, University of Colorado, NVIDIA, and the National Institutes of Health National Cancer Institute. Challenge participants were able to submit docker containers capable of implementing FL on three simulated medical facilities, each containing a unique large mammography dataset. The breast density FL challenge ran from June 15 to September 5, 2022, attracting seven finalists from around the world. The winning FL submission reached a linear kappa score of 0.653 on the challenge test data and 0.413 on an external testing dataset, scoring comparably to a model trained on the same data in a central location.