Machine Learning Estimation of Heterogeneous Treatment Effects with Instruments

TL;DR

This paper introduces a robust machine learning framework for estimating heterogeneous treatment effects using instruments, capable of handling unobserved confounders and leveraging advanced algorithms like neural networks and forests.

Contribution

It develops a loss-based estimation approach with Neyman orthogonality, allowing flexible use of modern ML methods and providing asymptotic inference for parametric effect models.

Findings

Successfully applied to TripAdvisor data with 4 million users.

Validated on synthetic and public datasets for schooling effects.

Achieved robust, flexible estimation of treatment effects with confidence intervals.

Abstract

We consider the estimation of heterogeneous treatment effects with arbitrary machine learning methods in the presence of unobserved confounders with the aid of a valid instrument. Such settings arise in A/B tests with an intent-to-treat structure, where the experimenter randomizes over which user will receive a recommendation to take an action, and we are interested in the effect of the downstream action. We develop a statistical learning approach to the estimation of heterogeneous effects, reducing the problem to the minimization of an appropriate loss function that depends on a set of auxiliary models (each corresponding to a separate prediction task). The reduction enables the use of all recent algorithmic advances (e.g. neural nets, forests). We show that the estimated effect model is robust to estimation errors in the auxiliary models, by showing that the loss satisfies a Neyman orthogonality criterion. Our approach can be used to estimate projections of the true effect model on simpler hypothesis spaces. When these spaces are parametric, then the parameter estimates are asymptotically normal, which enables construction of confidence sets. We applied our method to estimate the effect of membership on downstream webpage engagement on TripAdvisor, using as an instrument an intent-to-treat A/B test among 4 million TripAdvisor users, where some users received an easier membership sign-up process. We also validate our method on synthetic data and on public datasets for the effects of schooling on income.