Comparison Lift: Bandit-based Experimentation System for Online Advertising

TL;DR

Comparison Lift is a bandit-based experimentation system for online advertising at JD.com that improves testing efficiency and effectiveness, leading to higher click-through rates and cost savings.

Contribution

It introduces a novel bandit-based experimental algorithm that aligns testing with advertiser goals and reduces experimentation costs.

Findings

Increased click-through rates by 46% on average.

Generated 27% more clicks during testing compared to fixed A/B.

Used in over 1,500 experiments since 2019.

Abstract

Comparison Lift is an experimentation-as-a-service (EaaS) application for testing online advertising audiences and creatives at JD.com. Unlike many other EaaS tools that focus primarily on fixed sample A/B testing, Comparison Lift deploys a custom bandit-based experimentation algorithm. The advantages of the bandit-based approach are two-fold. First, it aligns the randomization induced in the test with the advertiser's goals from testing. Second, by adapting experimental design to information acquired during the test, it reduces substantially the cost of experimentation to the advertiser. Since launch in May 2019, Comparison Lift has been utilized in over 1,500 experiments. We estimate that utilization of the product has helped increase click-through rates of participating advertising campaigns by 46% on average. We estimate that the adaptive design in the product has generated 27% more clicks on average during testing compared to a fixed sample A/B design. Both suggest significant value generation and cost savings to advertisers from the product.