Improving reproducibility by controlling random seed stability in machine learning based estimation via bagging

TL;DR

This paper introduces a new method called adaptive cross-bagging that enhances the stability of machine learning predictions across different random seeds, improving reproducibility in debiased estimators.

Contribution

The paper formalizes seed stability, proves subbagging guarantees, and proposes adaptive cross-bagging to eliminate seed dependence in debiased machine learning.

Findings

Adaptive cross-bagging achieves targeted stability levels.

The method incurs minimal computational overhead.

It outperforms alternative approaches in numerical experiments.

Abstract

Predictions from machine learning algorithms can vary across random seeds, inducing instability in downstream debiased machine learning estimators. We formalize random seed stability via a concentration condition and prove that subbagging guarantees stability for any bounded-outcome regression algorithm. We introduce a new cross-fitting procedure, adaptive cross-bagging, which simultaneously eliminates seed dependence from both nuisance estimation and sample splitting in debiased machine learning. Numerical experiments confirm that the method achieves the targeted level of stability whereas alternatives do not. Our method incurs a small computational penalty relative to standard practice whereas alternative methods incur large penalties.