Assessing Estimate of CATE from Observational Data via an RCT Study

TL;DR

This paper introduces CAFE, a framework for evaluating the accuracy of CATE estimates from observational data using randomized trial evidence, applicable to various models including machine learning methods.

Contribution

The paper proposes a novel method, CAFE, for directly assessing the goodness-of-fit of CATE estimates with theoretical guarantees, accommodating diverse models and detecting unobserved confounders.

Findings

CAFE effectively detects lack of fit in CATE estimates.

The framework provides theoretical guarantees under null and alternative hypotheses.

Numerical studies demonstrate CAFE's utility in real-world scenarios.

Abstract

Conditional average treatment effects (CATEs) are increasingly estimated from observational data and used to guide policy and individualized treatment decisions. Before such estimates can be trusted in practice, their predictive fitness needs to be assessed, yet observational data alone offer limited opportunities for doing so. We propose CATE Assessment via Fitness Evaluation (CAFE), a formal framework for directly assessing the goodness-of-fit of a CATE estimate learned from observational data, rather than the full underlying outcome model, using evidence from a randomized trial. CAFE partitions the trial covariate space according to estimated propensity scores (or the like) and compares observationally derived conditional treatment effects with group-level experimental averages. The framework accommodates a broad class of CATE learners, including parametric models and flexible machine learning methods such as causal forest and boosting. We establish theoretical guarantees under both the null and alternative hypotheses, and introduce a maximum-type extension to improve sensitivity to localized lack of fit. When both randomized trial and observational data are available, we further develop a two-stage procedure to detect the existence of unobserved confounders. Extensive numerical studies show the utility of the CAFE approach when assessing observational-derived CATE estimates.