Out-of-sample scoring and automatic selection of causal estimators

TL;DR

This paper introduces novel out-of-sample scoring methods for causal estimators like CATE and IV models, enabling effective model selection and hyperparameter tuning in practical applications.

Contribution

It proposes new scoring approaches for causal models, especially for IV problems involving customer feature access, and provides an open source implementation integrating existing causal inference tools.

Findings

Scores reliably select models close to true impact on synthetic data.

Optimized scores improve model hyperparameter tuning.

Demonstrated effectiveness on real customer data from Wise.

Abstract

Recently, many causal estimators for Conditional Average Treatment Effect (CATE) and instrumental variable (IV) problems have been published and open sourced, allowing to estimate granular impact of both randomized treatments (such as A/B tests) and of user choices on the outcomes of interest. However, the practical application of such models has ben hampered by the lack of a valid way to score the performance of such models out of sample, in order to select the best one for a given application. We address that gap by proposing novel scoring approaches for both the CATE case and an important subset of instrumental variable problems, namely those where the instrumental variable is customer acces to a product feature, and the treatment is the customer's choice to use that feature. Being able to score model performance out of sample allows us to apply hyperparameter optimization methods to causal model selection and tuning. We implement that in an open source package that relies on DoWhy and EconML libraries for implementation of causal inference models (and also includes a Transformed Outcome model implementation), and on FLAML for hyperparameter optimization and for component models used in the causal models. We demonstrate on synthetic data that optimizing the proposed scores is a reliable method for choosing the model and its hyperparameter values, whose estimates are close to the true impact, in the randomized CATE and IV cases. Further, we provide examles of applying these methods to real customer data from Wise.