# Shrinkage Estimators in Online Experiments

## TL;DR

This paper introduces empirical Bayes shrinkage estimators for causal effect estimation in large-scale online experiments, especially addressing multiple treatment groups, improving accuracy and decision-making.

## Contribution

It develops consistent, low-bias shrinkage estimators that outperform traditional methods in online experiments with multiple treatments, enhancing sequential decision processes.

## Key findings

- Lower mean squared error compared to traditional estimators
- Retains frequentist coverage properties in most scenarios
- Improves treatment allocation efficiency in multi-armed bandit settings

## Abstract

We develop and analyze empirical Bayes Stein-type estimators for use in the estimation of causal effects in large-scale online experiments. While online experiments are generally thought to be distinguished by their large sample size, we focus on the multiplicity of treatment groups. The typical analysis practice is to use simple differences-in-means (perhaps with covariate adjustment) as if all treatment arms were independent. In this work we develop consistent, small bias, shrinkage estimators for this setting. In addition to achieving lower mean squared error these estimators retain important frequentist properties such as coverage under most reasonable scenarios. Modern sequential methods of experimentation and optimization such as multi-armed bandit optimization (where treatment allocations adapt over time to prior responses) benefit from the use of our shrinkage estimators. Exploration under empirical Bayes focuses more efficiently on near-optimal arms, improving the resulting decisions made under uncertainty. We demonstrate these properties by examining seventeen large-scale experiments conducted on Facebook from April to June 2017.