Mostly Harmless Simulations? Using Monte Carlo Studies for Estimator Selection

TL;DR

This paper critically evaluates Monte Carlo methods for estimator selection in treatment effect analysis, showing they are often ineffective and recommending robust comparison of multiple estimators instead.

Contribution

The paper demonstrates that recent Monte Carlo approaches for estimator selection are generally unreliable and advocates for using multiple estimators and robustness checks.

Findings

Monte Carlo methods often perform worse than random at bias minimization

They are somewhat better at mean squared error minimization

Bootstrap methods are at least as effective and often superior

Abstract

We consider two recent suggestions for how to perform an empirically motivated Monte Carlo study to help select a treatment effect estimator under unconfoundedness. We show theoretically that neither is likely to be informative except under restrictive conditions that are unlikely to be satisfied in many contexts. To test empirical relevance, we also apply the approaches to a real-world setting where estimator performance is known. Both approaches are worse than random at selecting estimators which minimise absolute bias. They are better when selecting estimators that minimise mean squared error. However, using a simple bootstrap is at least as good and often better. For now researchers would be best advised to use a range of estimators and compare estimates for robustness.