# Algorithm Adaptation Bias in Recommendation System Online Experiments

## TL;DR

This paper highlights the algorithm adaptation bias in online recommendation experiments, which can distort results and lead to suboptimal deployment decisions, emphasizing the need for improved evaluation methods.

## Contribution

It introduces and explains the algorithm adaptation bias, providing empirical evidence and discussing potential solutions to improve online recommendation experiment accuracy.

## Key findings

- Algorithm adaptation bias can significantly distort A/B test results.
- Empirical evidence shows bias favors large traffic variants.
- Addressing this bias can improve recommendation system deployment decisions.

## Abstract

Online experiments (A/B tests) are widely regarded as the gold standard for evaluating recommender system variants and guiding launch decisions. However, a variety of biases can distort the results of the experiment and mislead decision-making. An underexplored but critical bias is algorithm adaptation effect. This bias arises from the flywheel dynamics among production models, user data, and training pipelines: new models are evaluated on user data whose distributions are shaped by the incumbent system or tested only in a small treatment group. As a result, the measured effect of a new product change in modeling and user experience in this constrained experimental setting can diverge substantially from its true impact in full deployment. In practice, the experiment results often favor the production variant with large traffic while underestimating the performance of the test variant with small traffic, which leads to missing opportunities to launch a true winning arm or underestimating the impact. This paper aims to raise awareness of algorithm adaptation bias, situate it within the broader landscape of RecSys evaluation biases, and motivate discussion of solutions that span experiment design, measurement, and adjustment. We detail the mechanisms of this bias, present empirical evidence from real-world experiments, and discuss potential methods for a more robust online evaluation.