Reproducibility in Machine Learning for Health

TL;DR

This paper systematically evaluates reproducibility issues in machine learning for health, highlighting poor data and code accessibility, and proposes recommendations to improve research transparency and reliability.

Contribution

It provides a comprehensive assessment of reproducibility in ML4H research and offers actionable guidelines to enhance transparency and reproducibility in the field.

Findings

ML4H compares poorly to established ML fields in reproducibility.

Data and code accessibility are significantly lacking in ML4H papers.

Recommendations are proposed to improve reproducibility in the field.

Abstract

Machine learning algorithms designed to characterize, monitor, and intervene on human health (ML4H) are expected to perform safely and reliably when operating at scale, potentially outside strict human supervision. This requirement warrants a stricter attention to issues of reproducibility than other fields of machine learning. In this work, we conduct a systematic evaluation of over 100 recently published ML4H research papers along several dimensions related to reproducibility. We find that the field of ML4H compares poorly to more established machine learning fields, particularly concerning data and code accessibility. Finally, drawing from success in other fields of science, we propose recommendations to data providers, academic publishers, and the ML4H research community in order to promote reproducible research moving forward.