Understanding the Risks and Rewards of Combining Unbiased and Possibly Biased Estimators, with Applications to Causal Inference

TL;DR

This paper analyzes the trade-offs in combining unbiased and biased estimators for causal inference, introducing a new approach with better bias thresholds and providing bounds on mean squared error based on estimator variances.

Contribution

It offers a new estimator that balances bias and variance more effectively and provides theoretical bounds on its performance in the context of combining estimators.

Findings

The new estimator has a higher bias threshold than existing methods.

Simulation shows the proposed approach outperforms recent alternatives.

Theoretical bounds relate MSE to variances and covariances of estimators.

Abstract

Several problems in statistics involve the combination of high-variance unbiased estimators with low-variance estimators that are only unbiased under strong assumptions. A notable example is the estimation of causal effects while combining small experimental datasets with larger observational datasets. There exist a series of recent proposals on how to perform such a combination, even when the bias of the low-variance estimator is unknown. To build intuition for the differing trade-offs of competing approaches, we argue for examining the finite-sample estimation error of each approach as a function of the unknown bias. This includes understanding the bias threshold -- the largest bias for which a given approach improves over using the unbiased estimator alone. Though this lens, we review several recent proposals, and observe in simulation that different approaches exhibits qualitatively different behavior. We also introduce a simple alternative approach, which compares favorably in simulation to recent alternatives, having a higher bias threshold and generally making a more conservative trade-off between best-case performance (when the bias is zero) and worst-case performance (when the bias is adversarially chosen). More broadly, we prove that for any amount of (unknown) bias, the MSE of this estimator can be bounded in a transparent way that depends on the variance / covariance of the underlying estimators that are being combined.