High Precision Causal Model Evaluation with Conditional Randomization

TL;DR

This paper introduces a low-variance pairs estimator for evaluating causal models using conditional randomization, improving accuracy and reliability over traditional methods in real-world settings.

Contribution

The paper proposes a novel pairs estimator that reduces variance in causal error estimation under conditional randomization, enhancing model evaluation accuracy.

Findings

The pairs estimator achieves lower asymptotic variance compared to existing methods.

Empirical results show near-RCT performance in causal model evaluation.

The method simplifies evaluation without modifying the core IPW estimator.

Abstract

The gold standard for causal model evaluation involves comparing model predictions with true effects estimated from randomized controlled trials (RCT). However, RCTs are not always feasible or ethical to perform. In contrast, conditionally randomized experiments based on inverse probability weighting (IPW) offer a more realistic approach but may suffer from high estimation variance. To tackle this challenge and enhance causal model evaluation in real-world conditional randomization settings, we introduce a novel low-variance estimator for causal error, dubbed as the pairs estimator. By applying the same IPW estimator to both the model and true experimental effects, our estimator effectively cancels out the variance due to IPW and achieves a smaller asymptotic variance. Empirical studies demonstrate the improved of our estimator, highlighting its potential on achieving near-RCT performance. Our method offers a simple yet powerful solution to evaluate causal inference models in conditional randomization settings without complicated modification of the IPW estimator itself, paving the way for more robust and reliable model assessments.