Beyond Accuracy: An Empirical Study of Uncertainty Estimation in Imputation

TL;DR

This paper empirically evaluates how well different imputation methods estimate uncertainty across various datasets and missing data mechanisms, revealing that high accuracy does not always mean reliable uncertainty estimates.

Contribution

It systematically compares statistical, distribution alignment, and deep generative imputation methods in terms of uncertainty calibration and provides practical guidelines for their use.

Findings

Calibration often mismatched with accuracy

Stable configurations identified for reliable uncertainty

Trade-offs between accuracy, calibration, and runtime

Abstract

Handling missing data is a central challenge in data-driven analysis. Modern imputation methods not only aim for accurate reconstruction but also differ in how they represent and quantify uncertainty. Yet, the reliability and calibration of these uncertainty estimates remain poorly understood. This paper presents a systematic empirical study of uncertainty in imputation, comparing representative methods from three major families: statistical (MICE, SoftImpute), distribution alignment (OT-Impute), and deep generative (GAIN, MIWAE, TabCSDI). Experiments span multiple datasets, missingness mechanisms (MCAR, MAR, MNAR), and missingness rates. Uncertainty is estimated through three complementary routes: multi-run variability, conditional sampling, and predictive-distribution modeling, and evaluated using calibration curves and the Expected Calibration Error (ECE). Results show that accuracy and calibration are often misaligned: models with high reconstruction accuracy do not necessarily yield reliable uncertainty. We analyze method-specific trade-offs among accuracy, calibration, and runtime, identify stable configurations, and offer guidelines for selecting uncertainty-aware imputers in data cleaning and downstream machine learning pipelines.