Online activity prediction via generalized Indian buffet process models

TL;DR

This paper introduces a Bayesian nonparametric model based on the stable beta-scaled process for accurate, fast prediction of user activity in large-scale online experiments, enabling shorter testing periods with reliable uncertainty estimates.

Contribution

The paper presents a novel Bayesian nonparametric model for activity prediction in web experiments, capturing heavy-tailed behaviors and providing closed-form posterior and predictive distributions for rapid inference.

Findings

Improved forecasting accuracy over existing methods, especially with limited pilot data.

Enables shorter experiment durations while maintaining reliable uncertainty estimates.

Applicable to various large-scale, distribution-free prediction tasks.

Abstract

Online A/B experiments generate millions of user-activity records each day, yet experimenters need timely forecasts to guide roll-outs and safeguard user experience. Motivated by the problem of activity prediction for A/B tests at Amazon, we introduce a Bayesian nonparametric model for predicting both first-time and repeat triggers in web experiments. The model is based on the stable beta-scaled process prior, which allows for capturing heavy-tailed behaviour without strict parametric assumptions. All posterior and predictive quantities are available in closed form, allowing for fast inference even on large-scale datasets. Simulation studies and a retrospective analysis of 1,774 production experiments show improved accuracy in forecasting new users and total triggers compared with state-of-the-art competitors, especially when only a few pilot days are observed. The framework enables shorter tests while preserving calibrated uncertainty estimates. Although motivated by Amazon's experimentation platform, the method extends to other applications that require rapid, distribution-free prediction of sparse count processes.