Latent Stratification for Incrementality Experiments

TL;DR

This paper introduces a latent stratification model for incrementality experiments that improves the precision of treatment effect estimates by accounting for customer subgroups, reducing variance and decision errors.

Contribution

The paper presents a novel latent stratification approach that enhances ATE estimation accuracy in marketing experiments by modeling customer subgroups based on potential outcomes.

Findings

Variance of ATE reduced by 30-60% in real experiments.

Method performs well when treatment effects differ across latent strata.

Improves decision-making accuracy in marketing campaigns.

Abstract

Incrementality experiments compare customers exposed to a marketing action designed to increase sales to those randomly assigned to a control group. These experiments suffer from noisy responses which make precise estimation of the average treatment effect (ATE) and marketing ROI difficult. We develop a model that improves the precision by estimating separate treatment effects for three latent strata defined by potential outcomes in the experiment -- customers who would buy regardless of ad exposure, those who would buy only if exposed to ads and those who would not buy regardless. The overall ATE is estimated by averaging the strata-level effects, and this produces a more precise estimator of the ATE over a wide range of conditions typical of marketing experiments. Analytical results and simulations show that the method decreases the sampling variance of the ATE most when (1) there are large differences in the treatment effect between latent strata and (2) the model used to estimate the strata-level effects is well-identified. Applying the procedure to 5 catalog experiments shows a reduction of 30-60% in the variance of the overall ATE. This leads to a substantial decrease in decision errors when the estimator is used to determine whether ads should be continued or discontinued.