On Heavy-user Bias in A/B Testing

TL;DR

This paper analyzes how heavy-users can bias A/B testing results in online experimentation and proposes a re-sampling estimator to correct this bias, improving the reliability of short-term experiments.

Contribution

It provides a theoretical analysis of heavy-user bias in A/B testing and introduces a novel re-sampling estimator for bias correction.

Findings

Heavy-users significantly contribute to bias in A/B testing.

The proposed re-sampling estimator effectively reduces bias.

Theoretical analysis supports the estimator's validity.

Abstract

On-line experimentation (also known as A/B testing) has become an integral part of software development. To timely incorporate user feedback and continuously improve products, many software companies have adopted the culture of agile deployment, requiring online experiments to be conducted and concluded on limited sets of users for a short period. While conceptually efficient, the result observed during the experiment duration can deviate from what is seen after the feature deployment, which makes the A/B test result biased. In this paper, we provide theoretical analysis to show that heavy-users can contribute significantly to the bias, and propose a re-sampling estimator for bias adjustment.

Tables

Notation	Explanation
$Y_u\left(t\right)$	the observed outcome of user $u$ at day $t$.
$Z_u$	whether the user $u$ is in the treatment group.
$R_u\left(t\right)$	whether the user $u$ used the product at day $t$.
$t_u^0$	the first time user $u$ shows up, i.e. $\min \left\{t\mid R_u\left(t\right)=1\right\}$
${\tau }_u\left(t\right)$	the treatment effect for user $u$ at day $t$.
$c_u\left(t\right)$	the control outcome for user $u$ at day $t$.
$k$	duration of the experiment (day 1 to day $k$).
$N_T,N_C$	number of users in the treatment/control group.

Table 2. Table 1. Average biases and their standard errors

	Example 1	Example 2
Bias of original estimator	0.0220 (0.0023)	0.0373 (0.0020)
Bias of bias-adjusted estimator	-0.0022 (0.0026)	0.0132 (0.0023)
Bias of block-bootstrap estimator	0.0080 (0.0025)	0.0232 (0.0022)

Equations

$$P\{R_u(t)=a_t,{\tau }_u(t)\le b_t,c_u(t)\le c_t;t=1,\dots ,k\},$$

$$Y_u(t)=R_u(t)\{Z_u{\tau }_u(t)+c_u(t)\},$$

$$\frac{1}{k}\mathbb{E}\left\{\sum_{t=1}^{k}Y_{u}(t)\Big{|}Z_{u}=z\right\}\quad(z=1,0)$$

$${\Delta }_{\text{scaled}}=\mathbb{E}\left\{\frac{1}{k}\sum _{t=1}^k{R}_u(t){\tau }_u(t)\right\}.$$

$${\hat{\Delta }}_{\text{scaled}}=\frac{{\sum }_{u:Z_u=1}{\sum }_{t=1}^k{Y}_u(t)}{k{N}_T}-\frac{{\sum }_{u:Z_u=0}{\sum }_{t=1}^k{Y}_u(t)}{k{N}_C}.$$

$$\mathbb{E}\left(\widehat{\Delta}_{\textrm{scaled}}\right)=\mathbb{E}\left\{\frac{1}{k}\sum_{t=1}^{k}\tau_{u}(t)R_{u}(t)\Big{|}t_{u}^{0}\leq k\right\},$$

$${\Delta }_{\text{scaled}}=$$

$$\mathbb{E}\left({\hat{\Delta }}_{\text{scaled}}\right)=$$

$$=$$

$$f(p|t_u^0\le k)=$$

$$\underset{k\to \infty }{\lim }\mathbb{E}\left({\hat{\Delta }}_{\text{scaled}}\right)={\int }_0^1\tau (p)pf(p)dp={\Delta }_{\text{scaled}}.$$

$$\mathbb{E}\left({\hat{\Delta }}_{\text{scaled}}\right)-{\Delta }_{\text{scaled}}={\Delta }_{\text{scaled}}f(0)\cdot k^{-1}+O(k^{-2}).$$

$$\left|{\int }_0^{1}f(p)(1-p{)}^{k}dp-{\int }_0^{1}f(0)(1-p{)}^{k}dp\right|\le {\int }_0^{1}C\cdot p(1-p{)}^{k}dp,$$

$$\left|{\int }_0^{1}f(p)(1-p{)}^{k}dp-\frac{f(0)}{k+1}\right|\le \frac{1}{(k+1)(k+2)}=O(k^{-2}).$$

$$\hat{a}=k\cdot h(k)-(k-1)\cdot h(k-1).$$

$$\mathbb{E}{\hat{\Delta }}_{\text{jack}}-{\Delta }_{\text{scaled}}=O(k^{-2}).$$

$$\mathbb{E}{\hat{\Delta }}_{\text{jack}}=k\mathbb{E}{\hat{\Delta }}_{\text{scaled}}-\frac{(k-1)}{k}\sum _j\mathbb{E}{\hat{\Delta }}_{(j)}={\Delta }_{\text{scaled}}+O(k^{-2}).$$

$$Y_{u}(t)=\left\{\begin{array}[]{ll}1+p+N(0,0.01^{2})&\text{if treated}\\ 1+N(0,0.01^{2})&\text{otherwise}\end{array}\right.,$$

$$Y_{u}(t)=\left\{\begin{array}[]{ll}1+(1+\frac{1}{10\cdot U_{u}(t)})\cdot p+N(0,0.01^{2})&\text{if treated}\\ 1+N(0,0.01^{2})&\text{otherwise}\end{array}\right.,$$

Full text

External validity, jackknife, block bootstrap, causal inference

††copyright: rightsretained††conference: CIKM conference; November 2019; Beijing, China††journalyear: 2019

1. Introduction

A/B tests or online controlled experiments are used by a large number of software and technology companies (Hohnhold et al., 2015; Kharitonov et al., 2017) to evaluate changes to web services, desktop and mobile applications, and operating systems. In a typical online controlled experiment that is evaluating a new feature, users are randomly assigned to the treatment group (exposed to the new feature) or the control group (not exposed to the new feature) as they come online to use the software product or service. The assignment remains consistent throughout the experiment. During the experiment period, we collect telemetry from all users and compute metric for both groups. We conduct statistical tests to detect differences in metrics values between the treatment and control groups which are unlikely to be observed due to random chance. This establishes a causal relationship between the feature being tested and the measured changes in user behavior (Rubin, 2008; Imbens and Rubin, 2015).

One key touchstone of trustworthiness of experimentation is external validity (Campbell, 1957; Campbell and Stanley, 1963; Cook et al., 2002; Sabbaghi and Huang, 2018) – can results observed during an experiment period still hold when the new feature being tested is rolled out to the entire user population in the future? There can be multiple factors that affect external validity, such as the novelty effect and the weekday/weekend effect. While such factors are well recognized, there could be other neglected yet common effects that play an important role in determining external validity. In this paper, we highlight heavy-user bias, which describes the phenomenon that frequent users are more likely to be included in an experiment than infrequent ones, rendering the estimated average treatment effect biased. To our best knowledge, it has not been formally discussed in the existing data mining literature, and we hope that this paper can raise the community’s awareness on this important issue.

The remainder of the paper is organized as follows. Section 2 discusses the concepts of external validity and heavy user bias in more depth and introduces necessary notations and assumptions. Section 3 derives the closed-form expression of the heavy-user bias, proposes a bias-adjusted estimator based on jackknife (Quenouille, 1956; Tukey, 1958; Rao, 1965; Miller, 1974; Kunsch, 1989), and illustrates the performance of the bias-adjusted estimator via simulated examples. Section 4 concludes the paper by summarizing the progresses made in this on-going project, and discussing practical challenges and future research directions.

2. Preliminaries

2.1. External validity and heavy user bias

External validity, also known as generalizability (Stuart et al., 2011), refers to the problem generalizing the findings from the sample units included in the experiment to a larger inference population. External validity is an important problem in causal inference and several papers studied the external validity under a variety of different scenarios such as politics (Stuart et al., 2011) and education (Tipton et al., 2014). In the context of A/B tests, the external validity of A/B tests could be affected by a variety of factors, such as novelty/primacy effects (Sheinin et al., 2011) or weekday/weekend effects.

Heavy user bias is another important yet often overlooked factor that affects the external validity of A/B tests. To illustrate what is heavy user bias, let us consider a simple example. Suppose a website has two hundred users, half are heavy users who use the website every day, and the other half are light users who use the website with 50% probability each day. However, if an online experiment is run for just one day, there would be around 150 users using the website. The proportion of heavy users in the experiment sample will be 2/3. In other words, the proportion of heavy users for any A/B test is usually higher than that for the whole population. One simple way to mitigate heavy user bias would be to run the experiment long enough so that the proportion of heavy users and light users remain stable. However, under mild conditions, we show that the heavy user bias is usually inversely proportional to the length of the experiment $k$. This means when the experiment duration doubles, the heavy user bias is only reduced by half. Furthermore, long-term experiments are known to be prone to other critical issues (Dmitriev et al., 2016). Therefore, running experiments for longer periods might not be practical.

2.2. Notations and assumptions

The table below describes the notation we will use for the rest of the paper. To derive our theoretical result, we need to make the following assumptions. We will briefly comment why the introduced assumptions are reasonable in our application scenarios, and in Section 4 we will discuss how to further relax the assumptions.

Assumption 1 (stable unit treatment value assumption).

One user’s outcome is unaffected by other users’ treatment assignments. In other words, different users do not interfere with each other.

Assumption 2 (super population).

For each user, its behavior can be characterized as a series of triplets $\{R_{u}(t),\tau_{u}(t),c_{u}(t)\}_{t=1}^{k}$. We assume that this series for each user is an i.i.d. sample from a super population with a probability distribution $\Psi$:

[TABLE]

where $a_{t}\in\{0,1\}$, $b_{t},c_{t}\in\mathbb{R}$ for $t\in\{1,\ldots,k\}$.

Assumption 3 (incremental experiment assumption).

We assume that for each user $u$, the activity indicator $R_{u}(t)$ is independent of the treatment assignment $Z_{u}$.

Remark 1.

Under Assumption 1, the outcome of any user depends only on its own treatment assignment but not other users’ treatment assignment. This assumption is reasonable when users do not interact with each other, e.g., users of search engines and operating systems. However, it could break for users that can interact and communicate, such as users of social media. The latter scenarios are beyond the scope of this paper.

Assumption 3 implies that user’s visit $R_{u}(t)$ is not affected by whether a user is treated. In other words, we assume our experiment is incremental such that it does not change the frequency of users’ visits. This assumption could bring issues if a treatment significantly moves the number of days a user is active (i.e. uses the product). Therefore, it is important to test this assumption before analyzing the data using this framework. One plausible way would be to test whether the average active days per user is the same across treatment and control group. Based on our experience, most experiments do not affect the frequency of users’ visits significantly (Kohavi et al., 2014).

Under Assumptions 1–3, we can express the observed outcomes of the experimental units as

[TABLE]

which greatly facilitates the theoretical derivations going forward.

3. Methodology

3.1. Metric and point estimation

At the end of an A/B test, we compute metrics to estimate the impact of the treatment on user behavior and make ship decisions. For example, click-through rate (CTR) is a common metric for search engines to measure the effectiveness of online advertisements. In this paper, we focus on the scaled single average

[TABLE]

which are arguably the most common metric type in A/B testing111We will discuss other types of metrics in Section 4. Note that the expectation is calculated with respect to the data generating mechanism in (1), which means it is the average over all the users of the product. For this metric, the average treatment effect (ATE) is the difference between the metric for the treatment group and that for the control group:

[TABLE]

We can estimate $\Delta_{\textrm{scaled}}$ by the corresponding difference-in-sample-means derived from the observed data:

[TABLE]

Note that, in this equation, we only include $N_{T}+N_{C}$ users that appear during the experiment.

3.2. Heavy-user bias

We define the heavy-user bias of the estimator $\widehat{\Delta}_{\textrm{scaled}}$ estimating $\Delta_{\textrm{scaled}}$ as the difference between the expected value of the estimator and the estimand: $\mathbb{E}(\widehat{\Delta}_{\textrm{scaled}})-\Delta_{\textrm{scaled}}$.

Because only users who appear between day 1–$k$ are included in the experiment, the expectation of the point estimate of $\widehat{\Delta}_{\textrm{scaled}}$ is:

[TABLE]

The heavy-user bias is a (potentially complex) function of the data generating process in (1). To make the problem somewhat tractable and concrete, we propose a straightforward yet practical user behavior model, under which we derive the closed-form expressions of the heavy-user bias. We assume a fixed population, within which there exists user heterogeneity for both heavy and low activity frequencies and outcomes.

Model 1 (Fixed population with user heterogeneity).

We use the following model to reflect the heterogeneity on both user activity frequencies and outcomes:

• •

$R_{u}(t)$* for a user $u$ on day $t$ is i.i.d. from a Bernoulli random variable with success probability $p\sim f(\cdot)$.*

• •

The expectation of the treatment effect for user $u$ is $\mathbb{E}\tau_{u}(t)=\tau(p)$. It implies that the treatment effect could be different for users with different activity parameter $p$ but remains the same across all days.

• •

Similarly, the expected control outcome of a user is $\mathbb{E}c_{u}(t)=c(p)$.

As demonstrated in the following lemma, Model 1 allows us to derive the closed-form expressions for both $\Delta_{\textrm{scaled}}$ and $\mathbb{E}(\widehat{\Delta}_{\textrm{scaled}}),$ and therefore rigorously quantify the heavy-user bias.

Lemma 0.

Under Assumptions 1–3 and Model 1,

[TABLE]

and

[TABLE]

Proof.

First, (5) holds by the definition of Model 1. Second, based on (4),

[TABLE]

where $f(p|t_{u}^{0}\leq k)$ is the density of the user activity probability $p$ conditioned on $t_{u}^{0}\leq k$. Using the Bayes’ formula, we have

[TABLE]

By (8) and (9), we complete the proof. ∎

Remark 2.

Intuitively, the heavy-user bias arises because light users are less likely to show up during the experiment and therefore are under-represented. If we run the experiment long enough ($k\to\infty$), then

[TABLE]

However, for a finite period $k$, a user with activity probability $p$ has probability $1-(1-p)^{k}$ to show up during the experiment. That shows the proportion of heavy-users during the experiment are higher than that in the whole population.

With the help of Lemma 3.1, we can now present the main result of this paper.

Proposition 1.

Under Assumptions 1–3 and Model 1, if $f(\cdot),\tau(\cdot)$ has gradient and their gradients are continuous, then

[TABLE]

Proof.

Denote $f^{\prime}$ to be $f$’s gradient. Because $f^{\prime}$ is continuous on the compact set $[0,1],$ $f^{\prime}$ must be uniformly bounded on that set. That implies there exists a positive constant $C>0$ such that $|f(p)-f(0)|\leq C\cdot p$ for all $p\in[0,1].$ Therefore

[TABLE]

which implies that

[TABLE]

By (6) in Lemma 3.1 and (10), $\mathbb{E}\left\{\widehat{\Delta}_{\textrm{scaled}}\right\}=\Delta_{\textrm{scaled}}+f(0)\Delta_{\textrm{scaled}}k^{-1}+O(k^{-2}),$ which completes the proof. ∎

Remark 3.

When there is no extremely light users $f(0)=0$, it can be seen from the proposition that the first order bias of continuous analysis would be zero. Note that if a user has probability zero of showing up, it will never appear in the experiment. $f(0)$ should be thought of as the limit $\lim_{q\rightarrow 0}P(p\leq q)/q,$ which approximately represents users with very light activity. Based on our experience, for many online websites, the proportion of extremely light users is quite significant.

3.3. Bias-adjusted estimator

Having derived the heavy-user bias in Proposition 1, we next propose a bias-adjusted estimator to replace the difference-in-means estimator in (3). Our proposal is inspired by jackknife in classical literature(Tukey, 1958; Kunsch, 1989; Miller, 1974), which usually serves as a generic tool to correct first-order biases.

For any fixed experiment duration $k$, let $a$ be the true value of the outcome/metric of interest and $h(k)$ be an estimator of $a$. Assume the the heavy user bias of the estimator can be approximated by a function of $k$: $h(k)-a=b/k+O(k^{-2}),$ where $b$ is a parameter. The key insight here is that we can use two points $h(k-1)$ and $h(k)$ to get a better estimate of $a$ with almost no bias of order $O(k^{-1})$:

[TABLE]

For the scaled single average metric, the natural choice for $h(k)$ is the un-adjusted estimator $\widehat{\Delta}_{\textrm{scaled}}.$ To obtain $h(k-1)$, we can similarly calculate $\widehat{\Delta}_{\textrm{scaled}}$ using the data of first $k-1$ days. Although this estimate is unbiased, it does not use all the data at hand and can suffer from a large variance. To reduce its variance, we repeat the above procedure by excluding data from day $j=1,\ldots,k,$ and average the results. We summarize the above procedure in Algorithm 1.

Proposition 2.

Under Assumptions 1–3 and Model 1, the heavy-user bias of the bias-adjusted estimator in Algorithm 1 is

[TABLE]

Proof.

After excluding day $j,$ the remaining data can be viewed as from a $k-1$ day experiment. Thus we apply Proposition 1 to obtain the expectation of $\widehat{\Delta}_{(j)},$ the difference-in-means estimator on data excluding day $j:$ $\mathbb{E}\widehat{\Delta}_{(j)}=\Delta_{\textrm{scaled}}+\frac{1}{k-1}\Delta_{\textrm{scaled}}f(0)+O(k^{-2}).$ Therefore, the expectation of the bias-adjusted estimator is

[TABLE]

∎

3.4. Simulated examples

To demonstrate the advantages of the bias-adjusted estimator $\widehat{\Delta}_{\textrm{jack}}$ in Proposition 2, we present two simulated examples mimicking real-life A/B testing scenarios222We provide the source code of the simulations in https://github.com/shifwang/On-Heavy-user-Bias-in-AB-Testing. For both examples, the experiment lasts for 14 days, the treatment and control groups each contains 1000 units, and each unit uses the product with probability $p$ for day $t=1,\ldots,k$, where $p$ is generated from $\mathrm{Uniform}(0,1)$. The difference is how we generate the outcomes. To be specific, if user $u$ uses the product on day $t,$ for Example 1 we let

[TABLE]

and for Example 2 we let

[TABLE]

where $U_{u}(t)$ is the number of days the user $u$ used the product. For both examples, the ground truth $\Delta_{\textrm{scaled}}=1/3.$ While Example 1 is strictly under Model 1, Example 2 contains the novelty effect, which violates the assumptions in Model 1. By leveraging the two examples, we aim to examine both the accuracy and the robustness of the proposed bias-adjusted estimator.

For both examples, we repeat the data generation process 100 times. For each simulated data-set, we compute the original difference-in-means estimator $\widehat{\Delta}_{\textrm{scaled}},$ the bias-adjusted estimator $\widehat{\Delta}_{\textrm{jack}},$ and the block bootstrap estimator(Kunsch, 1989). We report the biases and the standard deviations of the three estimators in Table 1. The bias-adjusted estimator produces the smallest bias in both examples. Unfortunately, we do not have an answer why jackknife adjustment works better than bootstrap under our simulated settings.

4. Concluding Remarks

In this paper, we highlighted that the heavy-user bias could affect external validity significantly, and would like to raise the awareness of the data mining community on this issue. To be more specific, we demonstrated that the heavy-user bias exists in A/B testing due to the limited length of an experiment, and proposed a bias-adjusted estimator based on jackknife. Under the fixed population user heterogeneity model, we showed that jackknife estimators could correct the first order heavy-user bias. We conducted simulation studies to illustrate the advantages of the proposed methodology.

We summarize two lines of active research we have been conducting on this on-going project. First, besides (2) there are other types of metrics. For example, double average: $\Delta_{\textrm{double}}=\mathbb{E}\left\{\frac{\sum_{t=1}^{k}R_{u}(t)\tau_{u}(t)}{\sum_{t=1}^{k}R_{u}(t)}\right\}.$ Fortunately, we can leverage the same techniques to derive the heavy-user bias. Second, we applied the bias-adjusted estimator to several empirical data-sets and we found that, comparing with short term A/B testing results, our new estimator is closer to the long term A/B testing results.

To conclude the paper, we outline several future research directions to achieve the holy grail of ensuring external validity. First, it is important to understand the joint effect of multiple factors that affect external validity. The simulated examples showed that our proposed estimator provided a more accurate estimate in the presence of both the heavy user bias and the novelty effect. However, we might need to consider other possible effects in practice. Second, we can generalize the current methodology to study fairness in A/B testing, an important topic in machine learning and artificial intelligence (Corbett-Davies et al., 2017). Third, it would be interesting to extend the current study to the network setting with user interference.

5. Acknowledgement

The major work is done during Yu’s summer internship at ExP team at Microsoft Inc. in 2018. Partial supports are gratefully acknowledged from ARO grant W911NF1710005, ONR grant N00014-16-1-2664, NSF grants DMS-1613002, IIS 1741340, and CSoI under grant agreement CCF-0939370.