A cautionary tale on using imputation methods for inference in matched pairs design

TL;DR

This paper examines the impact of using machine learning-based imputation methods on statistical inference in matched pairs designs, revealing potential issues with inflated error rates and low power.

Contribution

It provides the first comprehensive analysis of the validity of machine learning imputation methods for inference in matched pairs, highlighting their limitations.

Findings

Machine learning imputation can inflate type-I error in small samples.

Imputation methods may reduce statistical power compared to complete case analysis.

The study includes extensive simulations and a real data example.

Abstract

Imputation procedures in biomedical fields have turned into statistical practice, since further analyses can be conducted ignoring the former presence of missing values. In particular, non-parametric imputation schemes like the random forest or a combination with the stochastic gradient boosting have shown favorable imputation performance compared to the more traditionally used MICE procedure. However, their effect on valid statistical inference has not been analyzed so far. This paper closes this gap by investigating their validity for inferring mean differences in incompletely observed pairs while opposing them to a recent approach that only works with the given observations at hand. Our findings indicate that machine learning schemes for (multiply) imputing missing values may inflate type-I-error or result in comparably low power in small to moderate matched pairs, even after modifying the test statistics using Rubin's multiple imputation rule. In addition to an extensive simulation study, an illustrative data example from a breast cancer gene study has been considered.