All models are wrong, but which are useful? Comparing parametric and nonparametric estimation of causal effects in finite samples

TL;DR

This study compares parametric and nonparametric methods for estimating causal effects in finite samples, finding nonparametric methods generally outperform parametric ones across diverse data scenarios.

Contribution

Introduces a Universal Monte-Carlo Simulation approach to evaluate causal estimators across numerous data-generating mechanisms, providing a comprehensive performance comparison.

Findings

Nonparametric estimators nearly always outperform parametric ones.

Parametric and nonparametric methods have similar bias at small sample sizes.

Nonparametric methods show better mean squared error and coverage in most cases.

Abstract

There is a long-standing debate in the statistical, epidemiological and econometric fields as to whether nonparametric estimation that uses data-adaptive methods, like machine learning algorithms in model fitting, confer any meaningful advantage over simpler, parametric approaches in real-world, finite sample estimation of causal effects. We address the question: when trying to estimate the effect of a treatment on an outcome, across a universe of reasonable data distributions, how much does the choice of nonparametric vs.~parametric estimation matter? Instead of answering this question with simulations that reflect a few chosen data scenarios, we propose a novel approach evaluating performance across thousands of data-generating mechanisms drawn from non-parametric models with semi-informative priors. We call this approach a Universal Monte-Carlo Simulation. We compare performance of estimating the average treatment effect across two parametric estimators (a g-computation estimator that uses a parametric outcome model and an inverse probability of treatment weighted estimator) and two nonparametric estimators (Bayesian additive regression trees and a targeted minimum loss-based estimator that uses an ensemble of machine learning algorithms in model fitting). We summarize estimator performance in terms of bias, confidence interval coverage, and mean squared error. We find that the nonparametric estimators nearly always outperform the parametric estimators with the exception of having similar performance in terms of bias and similar-to-slightly-worse performance in terms of coverage under the smallest sample size of N=100.