Randomized Experimental Design via Geographic Clustering

TL;DR

This paper introduces GeoCUTS, a novel algorithm for creating geographical clusters to minimize interference in web experimentations, validated through empirical analysis on Google Search data.

Contribution

The paper presents GeoCUTS, a new clustering algorithm that reduces interference in geographic experiments and a statistical framework to evaluate clustering effectiveness.

Findings

GeoCUTS performs comparably to hand-crafted regions.

The clustering minimizes user movement interference.

The framework effectively measures clustering quality.

Abstract

Web-based services often run randomized experiments to improve their products. A popular way to run these experiments is to use geographical regions as units of experimentation, since this does not require tracking of individual users or browser cookies. Since users may issue queries from multiple geographical locations, geo-regions cannot be considered independent and interference may be present in the experiment. In this paper, we study this problem, and first present GeoCUTS, a novel algorithm that forms geographical clusters to minimize interference while preserving balance in cluster size. We use a random sample of anonymized traffic from Google Search to form a graph representing user movements, then construct a geographically coherent clustering of the graph. Our main technical contribution is a statistical framework to measure the effectiveness of clusterings. Furthermore, we perform empirical evaluations showing that the performance of GeoCUTS is comparable to hand-crafted geo-regions with respect to both novel and existing metrics.