Cluster-Level Experiments using Temporal Switchback Designs: Precision Gains in Pricing A/B Tests at LATAM Airlines

TL;DR

This paper introduces switchback experimental designs for cluster-level testing in airline pricing, demonstrating significant precision improvements by reducing heterogeneity and noise through within-cluster contrasts.

Contribution

It provides a novel interpretation of switchback designs using Two-Way Fixed Effects and empirically evaluates their effectiveness in airline pricing experiments.

Findings

Switchback designs reduce standard errors by up to 67%.

Daily switching yields the largest precision gains over short periods.

Weekly switchbacks are a practical alternative with substantial improvements.

Abstract

Experimentation is central to modern digital businesses, but many operational decisions cannot be randomized at the user level. In such cases, cluster-level experiments, where clusters are usually geographic, come to the rescue. However, such experiments often suffer from low power due to persistent cluster heterogeneity, strong seasonality, and autocorrelated outcome metrics, as well as common shocks that move many clusters simultaneously. On an example of airline pricing - where policies are typically applied at the route level and thus the A/B test unit of analysis is a route - we study switchback designs to remedy these problems. In switchback designs, each cluster (route in our case) alternates between treatment and control on a fixed schedule, creating within-route contrasts that mitigate time-invariant heterogeneity and reduce sensitivity to low-frequency noise. We provide a unified Two-Way Fixed Effects interpretation of switchback experiments that makes the identifying variation explicit after partialling out route and time effects, clarifying how switching cadence interacts with temporal dependence to determine precision. Empirically, we evaluate weekly and daily switchback cadences using calibrated synthetic regimes and operational airline data from ancillary pricing. In our evaluations, switchbacks decrease standard errors by up to 67%, with daily switching yielding the largest gains over short horizons and weekly switching offering a strong and simpler-to-operationalize alternative.