Improving uplift model evaluation on RCT data

TL;DR

This paper analyzes the high variance of uplift model evaluation metrics on RCT data and proposes variance reduction methods based on outcome adjustment, demonstrating their effectiveness through theoretical and empirical results.

Contribution

It introduces variance reduction techniques for uplift evaluation metrics and provides theoretical and empirical evidence of their benefits on RCT data.

Findings

Variance reduction improves evaluation stability.

Methods outperform traditional metrics in noisy data.

Empirical validation on real-world datasets.

Abstract

Estimating treatment effects is one of the most challenging and important tasks of data analysts. In many applications, like online marketing and personalized medicine, treatment needs to be allocated to the individuals where it yields a high positive treatment effect. Uplift models help select the right individuals for treatment and maximize the overall treatment effect (uplift). A major challenge in uplift modeling concerns model evaluation. Previous literature suggests methods like the Qini curve and the transformed outcome mean squared error. However, these metrics suffer from variance: their evaluations are strongly affected by random noise in the data, which renders their signals, to a certain degree, arbitrary. We theoretically analyze the variance of uplift evaluation metrics and derive possible methods of variance reduction, which are based on statistical adjustment of the outcome. We derive simple conditions under which the variance reduction methods improve the uplift evaluation metrics and empirically demonstrate their benefits on simulated and real-world data. Our paper provides strong evidence in favor of applying the suggested variance reduction procedures by default when evaluating uplift models on RCT data.