A Large Scale Heterogeneous Treatment Effect Estimation Framework and Its Applications of Users' Journey at Snap

TL;DR

This paper introduces a large-scale industrial framework for estimating heterogeneous treatment effects using experimental data from hundreds of millions of Snapchat users, uncovering latent user characteristics and enabling effective targeted interventions.

Contribution

The paper presents a novel scalable framework for HTE estimation in industry, integrating experiment selection, base learner design, and incremental training to improve treatment effect stability and applicability.

Findings

Uncovered latent user characteristics influencing treatment effects

Achieved stable treatment effect estimates at scale

Online A/B test showed a sixfold improvement in key metrics

Abstract

Heterogeneous Treatment Effect (HTE) and Conditional Average Treatment Effect (CATE) models relax the assumption that treatment effects are the same for every user. We present a large scale industrial framework for estimating HTE using experimental data from hundreds of millions of Snapchat users. By combining results across many experiments, the framework uncovers latent user characteristics that were previously unmeasurable and produces stable treatment effect estimates at scale. We describe the core components that enabled this system, including experiment selection, base learner design, and incremental training. We also highlight two applications: user influenceability to ads and user sensitivity to ads. An online A/B test using influenceability scores for targeting showed an improvement on key business metrics that is more than six times larger than what is typically considered significant.