A Unified Framework for Inference with General Missingness Patterns and Machine Learning Imputation

TL;DR

This paper introduces a new statistical inference framework that effectively handles various missing data patterns using machine learning imputations under the realistic MAR assumption, improving accuracy and efficiency.

Contribution

It develops a novel method for valid inference with ML imputations across general missingness patterns under MAR, with theoretical guarantees and practical implementation strategies.

Findings

The proposed estimator is asymptotically normal.

It demonstrates efficiency gains over complete-case analysis.

Validated through extensive simulations and real data examples.

Abstract

Pre-trained machine learning (ML) predictions have been increasingly used to complement incomplete data to enable downstream scientific inquiries, but their naive integration risks biased inferences. Recently, multiple methods have been developed to provide valid inference with ML imputations regardless of prediction quality and to enhance efficiency relative to complete-case analyses. However, existing approaches are often limited to missing outcomes under a missing-completely-at-random (MCAR) assumption, failing to handle general missingness patterns (missing in both the outcome and exposures) under the more realistic missing-at-random (MAR) assumption. This paper develops a novel method that delivers a valid statistical inference framework for general Z-estimation problems using ML imputations under the MAR assumption and for general missingness patterns. The core technical idea is to stratify observations by distinct missingness patterns and construct an estimator by appropriately weighting and aggregating pattern-specific information through a masking-and-imputation procedure on the complete cases. We provide theoretical guarantees of asymptotic normality of the proposed estimator and efficiency dominance over weighted complete-case analyses. Practically, the method affords simple implementations by leveraging existing weighted complete-case analysis software. Extensive simulations are carried out to validate theoretical results. A real data example is provided to further illustrate the practical utility of the proposed method. The paper concludes with a brief discussion on practical implications, limitations, and potential future directions.