# On Heavy-user Bias in A/B Testing

## TL;DR

This paper analyzes how heavy-users can bias A/B testing results in online experimentation and proposes a re-sampling estimator to correct this bias, improving the reliability of short-term experiments.

## Contribution

It provides a theoretical analysis of heavy-user bias in A/B testing and introduces a novel re-sampling estimator for bias correction.

## Key findings

- Heavy-users significantly contribute to bias in A/B testing.
- The proposed re-sampling estimator effectively reduces bias.
- Theoretical analysis supports the estimator's validity.

## Abstract

On-line experimentation (also known as A/B testing) has become an integral part of software development. To timely incorporate user feedback and continuously improve products, many software companies have adopted the culture of agile deployment, requiring online experiments to be conducted and concluded on limited sets of users for a short period. While conceptually efficient, the result observed during the experiment duration can deviate from what is seen after the feature deployment, which makes the A/B test result biased. In this paper, we provide theoretical analysis to show that heavy-users can contribute significantly to the bias, and propose a re-sampling estimator for bias adjustment.