Adaptive Experimentation with Delayed Binary Feedback

TL;DR

This paper introduces an adaptive experimentation method designed for delayed binary feedback, improving efficiency over traditional and existing adaptive methods, and demonstrates its deployment in JD.com's online platform.

Contribution

The paper proposes a novel adaptive experimentation approach that estimates underlying objectives before they materialize, effectively handling delayed binary feedback.

Findings

The method outperforms other approaches in efficiency for delayed feedback.

It is robust across different experimental settings.

Successfully deployed in JD.com's online platform.

Abstract

Conducting experiments with objectives that take significant delays to materialize (e.g. conversions, add-to-cart events, etc.) is challenging. Although the classical "split sample testing" is still valid for the delayed feedback, the experiment will take longer to complete, which also means spending more resources on worse-performing strategies due to their fixed allocation schedules. Alternatively, adaptive approaches such as "multi-armed bandits" are able to effectively reduce the cost of experimentation. But these methods generally cannot handle delayed objectives directly out of the box. This paper presents an adaptive experimentation solution tailored for delayed binary feedback objectives by estimating the real underlying objectives before they materialize and dynamically allocating variants based on the estimates. Experiments show that the proposed method is more efficient for delayed feedback compared to various other approaches and is robust in different settings. In addition, we describe an experimentation product powered by this algorithm. This product is currently deployed in the online experimentation platform of JD.com, a large e-commerce company and a publisher of digital ads.