When Pattern-by-Pattern Works: Theoretical and Empirical Insights for Logistic Models with Missing Values

TL;DR

This paper introduces a Pattern-by-Pattern (PbP) approach for logistic models with missing data, providing theoretical guarantees and empirical comparisons with existing methods across various missing data scenarios.

Contribution

It proves that PbP accurately approximates Bayes probabilities under GPMM, even in MNAR settings, and compares its performance with imputation and EM methods.

Findings

PbP approximates Bayes probabilities under GPMM.

Mean imputation is effective for small samples, PbP for large samples.

Non-linear multiple imputation methods outperform in accuracy but are more costly.

Abstract

Predicting with missing inputs challenges even parametric models, as parameter estimation alone is insufficient for prediction on incomplete data. While several works study prediction in linear models, we focus on logistic models, where optimal predictors lack closed-form expressions. We prove that a Pattern-by-Pattern strategy (PbP), which learns one logistic model per missingness pattern, accurately approximates Bayes probabilities under a Gaussian Pattern Mixture Model (GPMM). Crucially, this result holds across standard missing data scenarios (MCAR and MAR) and, notably, in Missing Not at Random (MNAR) settings where standard methods often fail. Empirically, we compare PbP against imputation and EM methods across classification, probability estimation, calibration, and inference. Our analysis provides a comprehensive view of logistic regression with missing values. It reveals that mean imputation can be used as baseline for low sample sizes and PbP for large sample sizes, as both methods are fast to train and may have good performances in some settings. The best performances are achieved by non-linear multiple iterative imputation techniques that include the response label (Random Forest MICE with response), which are more computationally expensive.