Trimmed Match Design for Randomized Paired Geo Experiments

TL;DR

This paper introduces the Trimmed Match Design (TMD), a robust nonparametric method for designing reliable and cost-effective geo experiments to measure incremental ROAS, addressing challenges like small sample sizes and heavy-tailed responses.

Contribution

The paper proposes TMD, a novel systematic approach that extends Trimmed Match by integrating optimal subset pairing and sample splitting for geo experiment design.

Findings

TMD improves reliability of geo experiment results.

Simulation and case studies demonstrate TMD's effectiveness.

Addresses challenges of small sample sizes and heavy-tailed data.

Abstract

How to measure the incremental Return On Ad Spend (iROAS) is a fundamental problem for the online advertising industry. A standard modern tool is to run randomized geo experiments, where experimental units are non-overlapping ad-targetable geographical areas (Vaver & Koehler 2011). However, how to design a reliable and cost-effective geo experiment can be complicated, for example: 1) the number of geos is often small, 2) the response metric (e.g. revenue) across geos can be very heavy-tailed due to geo heterogeneity, and furthermore 3) the response metric can vary dramatically over time. To address these issues, we propose a robust nonparametric method for the design, called Trimmed Match Design (TMD), which extends the idea of Trimmed Match (Chen & Au 2019) and furthermore integrates the techniques of optimal subset pairing and sample splitting in a novel and systematic manner. Some simulation and real case studies are presented. We also point out a few open problems for future research.