# User Sentiment as a Success Metric: Persistent Biases Under Full Randomization

## TL;DR

This paper investigates the use of user sentiment as a metric in randomized A/B tests, addressing biases and proposing estimators to accurately measure treatment effects despite missing data and covariate influences.

## Contribution

It introduces consistent estimators for treatment effects in the context of user sentiment surveys, linking missing data issues with causal inference, and evaluates their performance through simulations.

## Key findings

- More complex models do not always outperform simpler ones.
- Identifies conditions for the existence of consistent estimators.
- Addresses persistent biases in user sentiment measurement.

## Abstract

We study user sentiment (reported via optional surveys) as a metric for fully randomized A/B tests. Both user-level covariates and treatment assignment can impact response propensity. We propose a set of consistent estimators for the average and local treatment effects on treated and respondent users. We show that our problem can be mapped onto the intersection of the missing data problem and observational causal inference, and we identify conditions under which consistent estimators exist. We evaluate the performance of estimators via simulation studies and find that more complicated models do not necessarily provide superior performance.