Optimal Data Integration and Adaptive Sampling for Efficient Treatment Effect Estimation

TL;DR

This paper introduces a shrinkage estimator and Bayesian adaptive design to efficiently estimate treatment effects in online advertising, reducing experimental costs while maintaining accuracy.

Contribution

It presents a novel combination of observational and experimental data integration with adaptive sampling, providing theoretical guarantees and practical benefits.

Findings

Achieves lower risk than existing methods

Reduces randomized experiments by half in real data

Provides theoretical guarantees including asymptotic normality

Abstract

This study addresses the challenge of estimating average treatment effects (ATEs) for advertising campaigns in online marketplaces where complete randomized experimentation is infeasible. We propose two key innovations: (1) a shrinkage estimator that optimally combines observational and experimental data without assuming smooth treatment effects across campaigns, and (2) a Bayesian adaptive experimental design framework that efficiently selects campaigns for randomized evaluation that minimizes cumulative risk. Our shrinkage estimator achieves lower risk compared to existing methods by balancing bias-variance tradeoffs, while our adaptive design significantly reduces the costs of campaign randomization. We establish theoretical guarantees including asymptotic normality and regret bounds. In an application to Amazon Ads data analyzing 2,583 campaigns, our approach achieves equivalent estimation precision while requiring only half of the randomized experiments needed by random sampling, the standard method widely used in practice today. The proposed method serves as a practical solution for marketplace platforms to efficiently measure advertising effectiveness while managing experimentation costs.