Construction and assessment of prediction rules for binary outcome in the presence of missing predictor data using multiple imputation: theoretical perspective and data-based evaluation

TL;DR

This paper develops and evaluates methods for calibrating and assessing binary outcome prediction rules with missing predictor data using multiple imputation, ensuring unbiased performance evaluation and practical implementation.

Contribution

It introduces two novel approaches for integrating multiple imputation with predictive calibration, compatible with existing software, and compares their effectiveness.

Findings

Little difference in accuracy between methods.

Significant reduction in variability of calibrated probabilities with the first approach.

Methods are applicable with current software for validation and imputation.

Abstract

We investigate the problem of calibration and assessment of predictive rules in prognostic designs when missing values are present in the predictors. Our paper has two key objectives which are entwined. The first is to investigate how the calibration of the prediction rule can be combined with the use of multiple imputation to account for missing predictor observations. The second objective is to propose such methods that can be implemented with current multiple imputation software, while allowing for unbiased predictive assessment through validation on new observations for which outcome is not yet available. To inform the definition of methodology, we commence with a review of the theoretical background of multiple imputation as a model estimation approach as opposed to a purely algorithmic description. We specifically contrast application of multiple imputation for parameter (effect) estimation with predictive calibration. Based on this review, two approaches are formulated, of which the second utilizes application of the classical Rubin's rules for parameter estimation, while the first approach averages probabilities from models fitted on single imputations to directly approximate the predictive density for future observations. We present implementations using current software which allow for validatory or cross-validatory estimation of performance measures, as well as imputation of missing data in predictors on the future data where outcome is by definition as yet unobserved. We restrict discussion to binary outcome and logistic regression throughout, though the principles discussed are generally applicable. We present two data sets as examples from our regular consultative practice. Results show little difference between methods for accuracy but substantial reductions in variation of calibrated probabilities when using the first approach.