User Sentiment as a Success Metric: Persistent Biases Under Full Randomization

TL;DR

This paper investigates the use of user sentiment as a metric in randomized A/B tests, addressing biases and proposing estimators to accurately measure treatment effects despite missing data and covariate influences.

Contribution

It introduces consistent estimators for treatment effects in the context of user sentiment surveys, linking missing data issues with causal inference, and evaluates their performance through simulations.

Findings

More complex models do not always outperform simpler ones.

Identifies conditions for the existence of consistent estimators.

Addresses persistent biases in user sentiment measurement.

Abstract

We study user sentiment (reported via optional surveys) as a metric for fully randomized A/B tests. Both user-level covariates and treatment assignment can impact response propensity. We propose a set of consistent estimators for the average and local treatment effects on treated and respondent users. We show that our problem can be mapped onto the intersection of the missing data problem and observational causal inference, and we identify conditions under which consistent estimators exist. We evaluate the performance of estimators via simulation studies and find that more complicated models do not necessarily provide superior performance.

Figures

Tables

Table 1. Table 1: Population level statistics based on GDP in Appendix  B .

Metric	Value
$E\left[\Delta \left(1\right)\right]$	$0.15$
$E\left[\Delta \left(0\right)\right]$	$0.09$
$E\left[Y\left(1\right)-Y\left(0\right)\right]$	$0$
$E\left[Y\left(1\right)-Y\left(0\right)|T=1,\Delta \left(1\right)=1\right]$	$0$
$E\left[Y\left(1\right)|\Delta \left(1\right)=1\right]-E\left[Y\left(0\right)|\Delta \left(0\right)=1\right]$	$0.07$

Table 2. Table 2: Performance of different ATE estimators when true confounders are fully observed. Estimators are Adversarial Balancing (AB), Doubly Robust (DR), Inverse Propensity Weighing (IPW), Naive mean comparison and Outcome Regression (OR). The naive estimator has the largest bias while AB estimator has the best MSE performance.

Estimator	Bias	MAE	MSE
AB	$2.8\text{×}{10}^{-2}$	$2.9\text{×}{10}^{-2}$	$1.5\text{×}{10}^{-3}$
DR	$-2.3\text{×}{10}^{-3}$	$6.1\text{×}{10}^{-2}$	$8.0\text{×}{10}^{-3}$
IPW	$5.7\text{×}{10}^{-2}$	$5.8\text{×}{10}^{-2}$	$6.2\text{×}{10}^{-3}$
Naive comparison	$7.1\text{×}{10}^{-2}$	$7.1\text{×}{10}^{-2}$	$5.8\text{×}{10}^{-3}$
OR	$8.8\text{×}{10}^{-4}$	$6.1\text{×}{10}^{-2}$	$8.1\text{×}{10}^{-3}$

Table 3. Table 3: Performance of different ATE estimators when noisy confounders are observed. Estimators are Adversarial Balancing (AB), Doubly Robust (DR), Inverse Propensity Weighing (IPW), Naive mean comparison and Outcome Regression (OR). In contrast to Table  2 , the performance of OR drops significantly while AB continues to provide a balance between variance and bias.

Estimator	Bias	MAE	MSE
AB	$3.4\text{×}{10}^{-2}$	$3.4\text{×}{10}^{-2}$	$1.8\text{×}{10}^{-3}$
DR	$6.2\text{×}{10}^{-3}$	$9.6\text{×}{10}^{-2}$	$2.0\text{×}{10}^{-2}$
IPW	$5.9\text{×}{10}^{-2}$	$6.0\text{×}{10}^{-2}$	$6.6\text{×}{10}^{-3}$
Naive comparison	$7.1\text{×}{10}^{-2}$	$7.1\text{×}{10}^{-2}$	$5.8\text{×}{10}^{-3}$
OR	$3.0\text{×}{10}^{-2}$	$1.0\text{×}{10}^{-1}$	$2.4\text{×}{10}^{-2}$

Table 4. Table 4: Performance of different ATETR estimators when true confounders are fully observed. Estimators are Adversarial Balancing (AB), Covariate Control (CC), Entrophy Balancing (EB), Inverse Propensity Weighing (IPW), Naive mean comparison and Outcome Regression (OR). EB has the best performance followed by simple CC estimator.

Estimator	Bias	MAE	MSE
AB	$3.2\text{×}{10}^{-2}$	$3.3\text{×}{10}^{-2}$	$1.7\text{×}{10}^{-3}$
CC	$7.7\text{×}{10}^{-3}$	$1.9\text{×}{10}^{-2}$	$7.8\text{×}{10}^{-4}$
EB	$4.3\text{×}{10}^{-3}$	$1.8\text{×}{10}^{-2}$	$7.1\text{×}{10}^{-4}$
IPW	$7.7\text{×}{10}^{-3}$	$1.9\text{×}{10}^{-2}$	$1.0\text{×}{10}^{-3}$
Naive comparison	$7.1\text{×}{10}^{-2}$	$7.1\text{×}{10}^{-2}$	$5.8\text{×}{10}^{-3}$
OR	$7.8\text{×}{10}^{-2}$	$7.8\text{×}{10}^{-2}$	$6.8\text{×}{10}^{-3}$

Table 5. Table 5: Performance of different ATETR estimators when noisy confounders are observed. Estimators are Adversarial Balancing (AB), Covariate Control (CC), Entrophy Balancing (EB), Inverse Propensity Weighing (IPW), Naive mean comparison and Outcome Regression (OR). Similar to the results in Table  4 , EB outperforms across all measures. Similar to Table  4 , simple CC estimator provides a comparable performance to EB.

Estimator	Bias	MAE	MSE
AB	$4.1\text{×}{10}^{-2}$	$4.1\text{×}{10}^{-2}$	$2.5\text{×}{10}^{-3}$
CC	$2.6\text{×}{10}^{-2}$	$2.8\text{×}{10}^{-2}$	$1.5\text{×}{10}^{-3}$
EB	$1.7\text{×}{10}^{-2}$	$2.3\text{×}{10}^{-2}$	$1.2\text{×}{10}^{-3}$
IPW	$-3.3\text{×}{10}^{-1}$	$3.1\text{×}{10}^{-2}$	$5.8\text{×}{10}^{-1}$
Naive comparison	$7.1\text{×}{10}^{-2}$	$7.1\text{×}{10}^{-2}$	$5.8\text{×}{10}^{-3}$
OR	$1.2\text{×}{10}^{-1}$	$1.2\text{×}{10}^{-1}$	$1.6\text{×}{10}^{-2}$

Equations

$$E[Y(1)-Y(0)],$$

$$E[Y(1)-Y(0)|T=1,\Delta (T)=1].$$

$${\Delta }_i(T_i)\perp Y_i(T_i)|X_i(T_i),$$

$$P({\Delta }_i(T_i)=1|X_i(T_i))>0$$

$$({\Delta }_i(0),{\Delta }_i(1))\perp Y_i(0)|X_i$$

$$P({\Delta }_i(0)=1|X_i)>0,P({\Delta }_i(1)=1|X_i)>0$$

$$E[Y(1)]$$

$$=E_X[E_Y[Y(1)|X,T=1,\Delta (1)=1]]$$

$${\theta }_1=|\{i:T(i)=1\}{|}^{-1}\sum _{i:T(i)=1}f(X_i)$$

$$ATETR$$

$$=E[Y(1)|T=1,\Delta (T)=1]$$

$$-E[Y(0)|T=1,\Delta (T)=1]$$

$$E_X[E_{X,\Delta (1)=1}[E_Y[Y(0)|\Delta (0)=1,X]|\Delta (1)=1,X]].$$

$$E[Y(1)|\Delta (1)=1]-E[Y(0)|\Delta (0)=1]\approx \mathrm{0.07.}$$

$$ATETR=E[Y(1)|T=1,\Delta (T)=1]-E[Y(0)|T=1,\Delta (1)=1].$$

$$E[Y(0)|T=1,\Delta (T)=1]$$

$$=E[E[Y(0)|\Delta (1)=1,\Delta (0)=1]|\Delta (1)=1]$$

$$=E[E[E[Y(0)|\Delta (1)=1,\Delta (0)=1,X]|\Delta (1)=1,X]]$$

$$=E[E[E[Y(0)|\Delta (0)=1,X]|\Delta (1)=1,X]].$$

$$X_1$$

$$S$$

$$Y$$

$$R$$

$$\Delta (T)$$

$$Z_1$$

$$Z_2$$

Full text

1 Introduction

Experimentation via A/B testing is widespread practice especially among technology companies [25, 14, 23]. A typical A/B test splits a population in two or more distinct groups at random, exposing each to different experiences. The experiment arm is then evaluated by a set of predefined metrics [23]. Engagement-based metrics are frequently used, and the entire process can be automated either by in-house infrastructure [23, 25] or by third party services [19].

Organizations have also measured user sentiment and attitudes at scale since the late 1980s [2, 7, 8], mainly driven by the relationship between consumer attitudes, long term loyalty, and revenue. Sentiment changes in an undesirable direction have also been shown to result in decreased engagement and lower long-term retention [8].

Improvements in survey technology have enabled faster and more targeted feedback. Historically, phone and mail surveys as well as in-person studies were the primary source of consumer data. Email-based surveys brought increased reach and decreased the cost of collecting user feedback. More recently, companies have brought surveys into UIs and started capturing user sentiment during the experience [17]. An example of a UI-based survey is given in Figure 1.

In recent years, companies have begun merging experimentation and user sentiment by utilizing sentiment as a success metric for product launches. This has become possible due to three major advantages of UI-based surveys over traditional survey mediums 1) mapping survey responses to treatment exposure is straightforward as surveys are attached to experiences, 2) capturing user sentiment in close temporal-proximity to the actual user experience increases validity of responses, 3) UI surveys have the potential to reach many users and have higher response rates compared to traditional channels [17].

Employing sentiment-based metrics in an A/B setting has its own challenges. First, unlike a typical engagement metric, sentiment data is incomplete in most cases. In particular, we have the ability to observe levels of engagement across all users in an A/B test111Clearly, some users may not engage, though this is an observation of no engagement and differs from survey non-response. while sentiment is measured through optional surveys to which not all users respond. Second, response behavior may be correlated with sentiment as well as treatment exposure. As a result, naive comparisons of sentiment-based metrics risk biased conclusions.

Accordingly, we focus on a scenario where user sentiment reported via optional surveys is used as a success metric for a fully randomized A/B test. There exists a set of user-level confounders which might impact the sentiment as well as the propensity to respond to the survey. Moreover, we allow treatment/control assignment to impact the probability of response. Our goal is to estimate both the population-level average treatment effect (ATE), and the local treatment effect on users that were treated and responded to the survey (ATETR).

2 Related Work and Our Contribution

Our setting relates to the well-known missing data and survey non-response problems [4, 12, 11]. Missing data literature focuses on the problem of estimating a population statistic from an incomplete dataset [15, 22, 13, 3]. In this scenario, there is no experimentation; only a single population with missing observations [13]. Typically, the data is assumed to be missing at random (MAR), i.e., missing data can be fully explained by a set of observed variables [22]. Under the MAR assumption, a rich set of consistent estimators have been proposed including outcome regression, inverse propensity weighing, and various doubly-robust methods [13, 3]. Though related, our problem departs from the traditional missing data problem because we have two distinct groups that are exposed to different experiences which impact both outcomes and the likelihood of missing data.

Our problem also intersects with causal inference, where one is interested in estimating the effect of a treatment using observational and/or quasi-experiment data [3, 20, 9, 5, 10, 21]. In the absence of a randomized controlled trial (RCT), the relationship between the treatment assignment and the outcome of interest may be confounded. The causal inference literature includes a wide range of methods to adjust for confounding in treated and non-treated populations based on a set of observed variables [3, 20, 5, 21]. While we rely heavily on causal inference, our setting also deviates from a typical causal problem. In particular, treatment assignment in our case is fully randomized, so there is no treatment selection bias. However, since responses are optional, there is still non-response bias.

We propose a framework at the intersection of missing data and causal inference to adjust for differential response bias in surveys in the A/B setting. We map ATE estimation into a missing data problem across two (or more) populations, and ATETR estimation into a causal inference problem by leveraging methods for estimating the average treatment effect on the treated (ATET). Additionally, we provide a simulation study using existing models to demonstrate a stylized application of our method. Finally, in the case of ATETR, we argue that a simple regression based covariate control may provide comparable performance to more complicated and resource heavy techniques.

3 Problem setup

Consider a scenario where user sentiment is used to evaluate a treatment. For the sake of simplicity, we assume that sentiment is collected via a single question222It is straightforward to expand our results to the case where sentiment is measured by a series of questions.. For instance, if we are interested in user satisfaction we may ask How satisfied are you with the overall experience?. A user might pick one of the following $\{$ Extremely Satisfied, Satisfied, Neither Satisfied Nor Dissatisfied, Dissatisfied, Extremely Dissatisfied $\}$[1, 24]. It is common practice to code favorable responses higher, e.g, ”Extremely Satisfied” is commonly coded as $5$ and ”Extremely Dissatisfied” is coded as $1$. Additionally, this scale can be simplified using a Top 2 Box score, i.e, $\{4,5\}$ (Satisfied, Extremely Satisfied) are mapped to $1$ and $\{1,2,3\}$ are mapped to [math] [16, 18]. We focus on Top 2 Box binary sentiment measures, though our proposed approach generalizes across different sentiment measures and coding practices. Furthermore, we restrict to the case where survey response is optional, which differentiates our problem from a typical RCT333If we could mandate survey response we would simply compare the averages of the treatment and control groups to calculate the average treatment effect..

Assume that there are $N$ individuals in a population, and the population is randomly split into treatment and control groups. We will assume that all of the individuals in the population are exposed to either a treatment or a control condition. $T_{i}$ is a dummy variable indicating whether the individual $i$ is exposed to treatment or control, i.e, $T_{i}=1$, if individual $i$ is exposed to treatment, and $T_{i}=0$ if individual is exposed to control. $(Y_{i}(1),Y_{i}(0))$ captures the (scalar) sentiment measure for individual $i$ under treatment and control conditions, respectively. Finally, $(\Delta_{i}(1),\Delta_{i}(0))$ are indicators for survey response behavior under treatment and control conditions. In other words, for a user assigned to condition $T_{i}$, $\Delta_{i}(T_{i})$ will be equal to $1$ if the user responds to the survey.

We are interested in estimating two distinct quantities: the average treatment effect on the overall population (ATE), i.e:

[TABLE]

and the average treatment effect on individuals who are exposed to the treatment and respond to the survey (Average Treatment Effect on Treated and Responded - ATETR), i.e:

[TABLE]

The former resembles an engagement metric, which is observed across all individuals, by aiming to provide population level impact. Given the wide adoption of engagement-based metrics for product evaluations, comparability and consistency with such metrics is an important factor. However, stricter assumptions are required to extrapolate from the observed responses to the overall population. Such assumptions may be problematic given that average online surveys have a response rate lower than 10$\%$ [6]. While the latter measure does not require estimating the sentiment of survey non-respondents, it is more difficult to compare to engagement type metrics since the ATETR estimates a local effect rather than a population effect.

Going forward, we utilize two assumptions depending on the measure of interest.

Assumption 1** (Missing At Random)**

There exists a set of observed variables $X_{i}(T_{i})$, $T_{i}\in\{0,1\}$ such that:

[TABLE]

$\forall~{}i\in\mathcal{N}=\{1,\ldots,N\}$.

The first equation of Assumption 1 is known as Missing At Random (MAR), and says that systematic differences between the missing and observed values can be entirely explained by a set of observed variables [22, 15, 13]. The second equation of the assumption requires that the probability of observing complete data is always positive for a given individual $i$. Since $X_{i}(T_{i})$ is a function of the assignment, i.e, $T_{i}$, it is possible that different sets of variables can capture confoundedness in treatment and control groups. We present the causal graph representing Assumption 1 in Figure 2. The equivalent of the causal graph for $T=0$ can be constructed simply by replacing $(1)$’s with $(0)$’s.

Assumption 2

There exists a set of observed variables $X_{i}$ such that:

[TABLE]

The first equation of Assumption 2 requires the joint survey response behaviors to be independent of the sentiment under the control condition given the confounders. We require this assumption to estimate the ATETR, $E[Y(1)-Y(0)|T=1,\Delta(T)=1]$. Specifically, we want to estimate how those who were assigned to the treatment group and responded to the survey ($Y_{i}(0),\Delta_{i}(0)=1$) would have responded under control. Note that Assumption 2 does not impose any constraints on the conditional independence of $Y_{i}(1)$ since we observe the responses from the users who were treated and responded to the survey, thus we only need to estimate the counterfactual sentiment for treated and respondent users under control.

We show the causal graph representing Assumption 2 in Figure 3.

4 Balancing Treatment and Control

We propose mechanisms to estimate the measures of interest: ATE and ATETR.

4.1 Sentiment Average Treatment Effect (ATE)

We are interested in estimating the Average Treatment Effect on user sentiment over the entire population. Our first results enables parallelization of estimation of ATE by mapping it onto two missing data problems:

Lemma 1

Let $\theta_{1}$ denote a consistent estimator of $E[Y(1)|T=1]$, and $\theta_{0}$ denote a consistent estimator of $E[Y(0)|T=0]$. Then, $\theta_{1}-\theta_{0}$ is a consistent estimator of ATE.

The proof of the lemma is straightforward, and follows from the linearity of expectation, linearity of convergence in probability, and randomized treatment assignment i.e, $E[Y(1)|T=1]=E[Y(1)|T=0]=E[Y(1)]$. Lemma 1 suggests that one can focus on the treatment and control groups separately and treat each case as an independent missing data problem. Independence results from the fully randomized treatment/control assignment even though survey measures suffer from response bias. Lemma 1 is powerful since it enables parallelization of the estimation.

While we have mapped our ATE estimation into two independent missing data problems, we need further assumptions in order to provide consistent estimators, i.e., $\theta_{1},\theta_{0}$. In particular, we rely on Assumption  1 (also known as MAR [3]) which guarantees consistency for a wide range of estimators including outcome regression (OR), inverse propensity weighting (IPW), and adversarial balancing (AB) estimators444Additional assumptions for model specifications are required for parametric models..

As discussed in detail in [3], Assumption 1 enables us to infer sentiment measures for users who have not responded to the survey in the treatment (control) group by simply utilizing the responses of the users who have responded to the survey in the treatment (control) group.

As an example, we focus on the OR estimator and demonstrate how $\theta_{1},\theta_{0}$ can be calculated. We start with a set of equalities:

[TABLE]

where the first line follows from law of iterated expectation, and the second from Assumption 1 and the fact that treatment assignment is random. Equation (7) shows that building an estimator $E[Y_{i}(1)|X_{i}]=f(X_{i})$ trained on observed responses only ($\{i~{}|T_{i}=1,\Delta_{i}(1)=1\}$) provides a consistent estimator for $E[Y(1)]$ provided that $f(.)$ is correctly specified.

[TABLE]

is the estimator of interest, where $|.|$ is the cardinality of the set. $\theta_{0}$ can be calculated in a similar fashion. Interested readers can refer to [3] for detailed discussions.

We implemented several estimators (including outcome regression) for comparison purposes, and discuss performance in Section 5.

4.2 Sentiment Average Treatment Effect on Treated and Responded (ATETR)

Now we estimate the Average Treatment Effect on Treated and Responded given in Equation (2).

Similar to its counterpart in the causal inference literature, i.e., Average Treatment Effect on Treated (ATET), ATETR requires reconstructing counterfactuals for a subset of users. In our case, the subset is those who were assigned to the treatment group and chose to respond to the survey.

There is a consistent estimator for ATETR, defined as follows:

Lemma 2

Given the definition of ATETR in Equation (2) and Assumption 2,

[TABLE]

Moreover, $|\{i:T_{i}=1,\Delta_{i}(1)=1\}|^{-1}\sum_{i:T(i)=1,\Delta(i)=1}Y_{i}(1)$ is a consistent estimator of Equation (8), and Equation (9) is equal to:

[TABLE]

Proof of Lemma 2 is given in Appendix A.

As an example, we will once again focus on the OR estimator and demonstrate how ATETR can be calculated.

Equation (8) can simply be estimated by taking the average sentiment among users that were assigned to the treatment group and responded to the survey ($\{i|T_{i}=1,\Delta_{i}(1)=1\}$). Equation (9) (or equivalently Equation (10)) can be estimated by a 3 step process:

Focus on survey respondents under the control assignment, i.e., $\{i~{}|T_{i}=0,\Delta_{i}(0)=1\}$, and train a model for $E[Y_{i}(0)|X_{i}]=g(X_{i})$. 2. 2.

Construct counterfactual responses for users that were assigned to the treatment and responded to the survey, $E[Y_{i}(0)|T_{i}=1,\Delta_{i}(1)]$ using $g(X_{i})$, $\forall~{}i\in\{i~{}|T_{i}=1,\Delta_{i}(1)=1\}$. 3. 3.

Estimate $E[Y(1)|T=1,\Delta(1)=1]$ as ${|{i:T(i)=1,\Delta_{i}(1)=1}|^{-1}}\sum_{i:T_{i}=1}g(X_{i})$

In other words, training a model on the control group and using the trained model to predict counterfactuals over the treatment group provides a consistent estimator. Finally, the difference between the two provides a consistent estimator for ATETR555Additional assumptions for model specifications are required..

Similar to ATE, we implemented several estimators (including outcome regression) for comparison purposes, and discuss performance in Section 5.

5 Simulations

We applied our methodology to a simulated dataset. A causal graph representing the generative process is given in Figure 4. Latent variables ($X=[X_{1},X_{2}]$) generate both individual sentiment $Y$ and response behavior under the treatment and control conditions $\Delta(1),\Delta(0)$. User sentiment $Y$ is the same across treatment/control groups. However, response behaviors $\Delta(0),\Delta(1)$ differ under treatment vs. control and correlate with sentiment, so observed sentiment differs.

Details of the data generation process (DGP) are given in Appendix B. There are $4$ important points regarding the DGP:

Both Assumptions 1 and 2 hold. 2. 2.

There is no difference in the population level sentiment, $E[Y(1)]=E[Y(0)]$. 3. 3.

There are differences in the observed sentiment:

[TABLE] 4. 4.

Response rates under the treatment and control are also different: $E[\Delta(1)]\approx 0.15$, and $E[\Delta(0)]\approx 0.09$. These numbers correspond to $15\%$ and $9\%$ survey response rates, which are consistent with lower response rates typically observed for online surveys.

We run $5K$ Monte Carlo simulations with $N=10K$ ($\sim 5K$ each in treatment and control). $10K$ is used as the per simulation population size to reflect the small sample sizes typically observed during sentiment studies. Summary statistics are available in Table 1.

5.1 Sentiment Average Treatment Effect (ATE)

We implemented four different estimators and compared their performance under two different scenarios.

In scenario 1, we assume that we can observe the actual values of the confounding factors, i.e., $X_{1},X_{2}$. This allows us to correctly specify a model to adjust for all of the confounding variables, and Assumption 1 holds.

Our first estimator is a naive mean comparison among respondents. By definition this is an inconsistent estimator and does not take into account observed confounders. The remaining estimators are based on Lemma 1, and use the observed covariates to correct for the underlying bias. We implemented outcome regression (OR) [3], inverse propensity weighing (IPW) [13], a doubly robust (DR) estimator based on OR and IPW [3], and an adversarial balancing based estimator (AB) using Logistic Regression as the classifier [20].

We measured the performance of each estimator via bias, median absolute error (MAE), and mean squared error (MSE)666Bias = $E[ATE-\hat{ATE}]$, MAE = $Median(|ATE-\hat{ATE}|)$, and MSE = $E[(ATE-\hat{ATE})^{2}]$ as discussed in [13]. Results are given in Table 2. See Figure 5 for the distribution of estimates.

Naive comparison incorrectly reports a $0.07$ ($7$ percentage points) difference between treatment and control. Variation in the naive comparison is due to the stochasticity of the simulation. The OR estimator has the smallest bias, despite the fact that it suffers from higher variance. IPW has a large bias comparable to the naive estimator. The DR estimator’s performance falls in between OR and IPW.

In scenario 2, we relax our assumption on the ability to observe the true confounders. Instead, we assume that we can only observe transformations of the true confounders as given in Eq. (15)-(16). This corresponds to a case where the researcher cannot directly observe $X$, but can still access a set of variables that are related to the confounders themselves (and thus will likely misspecify the model). Results and boxplots are given in Table 3 and Figure 6, respectively.

Our first observation is the dramatic drop in the performance of OR. The estimator performs the worst in both bias and MSE. DR achieves the best performance in this case with respect to bias, however it once again suffers from large variance. The AB estimator seems to provide a balance between bias and variance.

5.2 Sentiment Average Treatment Effect on Treated and Responded (ATETR)

Similar to the ATE case, we have implemented five different estimators under two different scenarios.

In scenario 1, we assume that we can observe the actual values of the confounding factors, i.e., $X_{1},X_{2}$. Our first estimator is a naive mean comparison among respondents. The naive estimator will return the same estimate for ATE and ATETR. We implemented outcome regression (OR) [3], inverse propensity weighing (IPW) [13], covariate control (CC), an adversarial balancing based estimator777We use logistic regression for the discriminator. (AB) [20], and an entropy balancing888We utilized first moment based balancing for EB estimator. based method (EB) [9]. The main difference between ATE and ATETR estimation is the latter utilizes data among respondents only.

Results and boxplots are given in Table 4 and Figure 7, respectively. In our tests, the OR estimator has the largest bias. The estimator is trained using control-assigned survey respondents, and counterfactuals for the treatment assigned respondents are generated using the trained model. In our setting, the OR estimator consistently underestimates counterfactuals for the treatment-assigned respondents, resulting in a bias that is comparable with the naive estimator. CC, EB and IPW have small biases; EB and CC estimators have the best MSE performance across all estimators.

In scenario 2, we relax our assumption on our ability to observe the true confounders, and we assume that we can only observe transformations of the true confounders as given in Eq. (15)-(16). Results and boxplots are given in Table 5 and Figure 8, respectively.

CC suffers from a dramatic drop in performance when compared to the fully observed covariates scenario. We also observe dramatic bias and higher levels of variance in IPW as well as a large number of estimates with extreme values. The performance of the AB estimator seems comparable between the two scenarios. While the EB estimator suffers from increased bias, its bias and MSE are nevertheless the lowest.

6 Conclusion

In this paper, we focused on a scenario where user reported sentiment is used as a success metric for a fully randomized A/B test. Sentiment is assumed to be collected via a survey, and users have the option to not respond to the survey.

In the case of Average Treatment Effect (ATE), we showed that our problem can be broken down into two independent missing data problems. In other words, we can focus on treatment and control sets separately, and estimate population-level sentiment for both groups. Then we show that a simple difference between the two quantities provides a consistent estimator of the ATE. This also enables parallel execution for more efficient computation.

We simulate the problem with both fully observed and transformed confounders, observing significant performance decreases between two settings, highlighting the potential sensitivity of causal inference techniques to model misspecification. The adversarial balancing estimator seems to provide a desirable balance between bias and variance in the transformed confounders scenario.

In the case of Average Treatment Effect on Treated and Responded (ATETR), we showed that one can focus on respondent data only. Our problem can be mapped onto a standard causal analysis setting, i.e., matching respondents in the control group to respondents in the treatment group using the observed covariates. This enabled us to leverage existing techniques in the causal analysis literature.

Similarly to ATE, we observe significant performance decreases between the two ATETR settings. In particular, IPW suffers from dramatic performance degradation (potentially due to extreme values of predicted propensities). Interestingly, a simple regression-based covariate control (CC) was one of the best estimators for the fully observed case, suggesting more complicated techniques do not necessarily provide superior performance.

Our novel framework combined with simple regression or balancing techniques has the potential to be integrated with existing experimentation platforms to provide robust, efficient, and automatic impact calculations for sentiment metrics.

Appendix A Proof of Lemma 2

First recall that ATETR is defined as $E[Y(1)-Y(0)|T=1,\Delta(T)=1]$ in Equation (2). By the linearity of the expectation operator:

[TABLE]

A simple average over the set of users who are assigned to the treatment ($T=1$) and responded to the survey $\Delta(1)=1$ is a consistent estimator of the first quantity, i.e., $E[Y(1)|T=1,\Delta(T)=1]$. This follows from the fact that assignment is random.

If we focus on the second term, the following set of equalities hold:

[TABLE]

Equation (11) follows from the fact that treatment/control groups are randomly assigned, Equation (12) and  (13) hold due to the law of iterated expectations, Equation (14) follows from Assumption 2.

Appendix B Data Generating Process for simulations in Section 5

Given our causal graph in Figure 4, we draw Monte Carlo simulations using the following data generating process (DGP):

[TABLE]

where $\text{Exp}(\lambda)$ is an exponential distribution with rate $\lambda$, $\text{Bin}(\theta)$ is a binomial distribution with probability $\theta$, $T$ is a treatment indicator ($T=1$ for treatment and $T=0$ for control), $Y$ is the sentiment, and $\Delta(T)$ is an indicator of whether a response was received under treatment $T$. Our DGP satisfies both Assumptions 1 and 2 since $X=[X_{1},X_{2}]$ confound both response rates and sentiment. True sentiment $Y$ is the same across treatment and control since parameter $S$ is independent of the treatment indicator $T$. Response rates, however, are a function of both the confounders $X$ and the treatment indicator $T$. Therefore, the observed sentiment for the two groups is different.

In some of our simulations, we relax the observability on the true confounders $X_{1},X_{2}$ to study the effects of model misspecification. We instead assume that we can observe functions of the confonfounders, $Z_{1},Z_{2}$ where:

[TABLE]

This corresponds to a scenario where a researcher cannot directly observe the confounders but has access to a set of proxy variables for the confounders.