# Measuring Average Treatment Effect from Heavy-tailed Data

## TL;DR

This paper addresses the challenge of estimating average treatment effects in heavy-tailed data by proposing robust statistical methods like winsorization and Huber regression, combined with orthogonal learning to reduce variance and bias.

## Contribution

It introduces a novel framework combining robust regression and orthogonal learning to improve treatment effect estimation in heavy-tailed, large-scale online data.

## Key findings

- Huber regression reduces variance in treatment effect estimates.
- Orthogonal learning with residuals improves bias control.
- Proposed methods outperform traditional approaches in large-scale Ecommerce data.

## Abstract

Heavy-tailed metrics are common and often critical to product evaluation in the online world. While we may have samples large enough for Central Limit Theorem to kick in, experimentation is challenging due to the wide confidence interval of estimation. We demonstrate the pressure by running A/A simulations with customer spending data from a large-scale Ecommerce site. Solutions are then explored. On one front we address the heavy tail directly and highlight the often ignored nuances of winsorization. In particular, the legitimacy of false positive rate could be at risk. We are further inspired by the idea of robust statistics and introduce Huber regression as a better way to measure treatment effect. On another front covariates from pre-experiment period are exploited. Although they are independent to assignment and potentially explain the variation of response well, concerns are that models are learned against prediction error rather than the bias of parameter. We find the framework of orthogonal learning useful, matching not raw observations but residuals from two predictions, one towards the response and the other towards the assignment. Robust regression is readily integrated, together with cross-fitting. The final design is proven highly effective in driving down variance at the same time controlling bias. It is empowering our daily practice and hopefully can also benefit other applications in the industry.