Evaluating Uplift Modeling under Structural Biases: Insights into Metric Stability and Model Robustness

TL;DR

This paper systematically evaluates uplift modeling in biased real-world data, highlighting the importance of metric stability and model robustness, and introduces a benchmarking framework using semi-synthetic data to assess performance under structural biases.

Contribution

It introduces a semi-synthetic benchmarking framework for uplift models under biases and provides insights into model robustness and metric stability in such settings.

Findings

TARNet shows notable robustness to biases.

Uplift prediction and targeting are distinct objectives.

Metrics aligned with ATE offer more stable model rankings.

Abstract

In personalized marketing, uplift models estimate incremental effects by modeling how customer behavior changes under alternative treatments. However, real-world data often exhibit biases - such as selection bias, spillover effects, and unobserved confounding - which adversely affect both estimation accuracy and metric validity. Despite the importance of bias-aware assessment, a lack of systematic studies persists. To bridge this gap, we design a systematic benchmarking framework. Unlike standard predictive tasks, real-world uplift datasets lack counterfactual ground truth, rendering direct metric validation infeasible. Therefore, a semi-synthetic approach serves as a critical enabler for systematic benchmarking, effectively bridging the gap by retaining real-world feature dependencies while providing the ground truth needed to isolate structural biases. Our investigations reveal that: (i) uplift targeting and prediction can manifest as distinct objectives, where proficiency in one does not ensure efficacy in the other; (ii) while many models exhibit inconsistent performance under diverse biases, TARNet shows notable robustness, providing insights for subsequent model design; (iii) evaluation metric stability is linked to mathematical alignment with the ATE, suggesting that ATE-approximating metrics yield more consistent model rankings under structural data imperfections. These findings suggest the need for more robust uplift models and metrics. Code will be released upon acceptance.