Trustworthy assessment of heterogeneous treatment effect estimator

TL;DR

This paper introduces a new method for comparing heterogeneous treatment effect estimators by focusing on their relative error and incorporating uncertainty quantification, leading to more reliable and powerful assessments.

Contribution

It develops a relative error estimator based on the efficient influence function, providing a robust and more informative evaluation framework for HTE estimators.

Findings

The relative error estimator is less sensitive to nuisance estimation errors.

Confidence intervals for the relative error are narrower and more reliable.

The method effectively identifies better HTE estimators in benchmark datasets.

Abstract

Accurate heterogeneous treatment effect (HTE) estimation is essential for personalized recommendations, making it important to evaluate and compare HTE estimators. Traditional assessment methods are inapplicable due to missing counterfactuals. Current HTE evaluation methods rely on additional estimation or matching on test data, often ignoring the uncertainty introduced and potentially leading to incorrect conclusions. We propose incorporating uncertainty quantification into HTE estimator comparisons. In addition, we suggest shifting the focus to the estimation and inference of the relative error between methods rather than their absolute errors. Methodology-wise, we develop a relative error estimator based on the efficient influence function and establish its asymptotic distribution for inference. Compared to absolute error-based methods, the relative error estimator (1) is less sensitive to the error of nuisance function estimators, satisfying a "global double robustness" property, and (2) its confidence intervals are often narrower, making it more powerful for determining the more accurate HTE estimator. Through extensive empirical study of the ACIC challenge benchmark datasets, we show that the relative error-based method more effectively identifies the better HTE estimator with statistical confidence, even with a moderately large test dataset or inaccurate nuisance estimators.