Representation-Aware Experimentation: Group Inequality Analysis for A/B Testing and Alerting

TL;DR

This paper presents methods for detecting and analyzing unintended consequences of A/B tests on group representation, focusing on gender equality, using novel statistics and causal trees, with practical application at LinkedIn.

Contribution

It introduces the DER statistic for measuring deviation from equal representation and adapts causal trees for heterogeneous treatment effect analysis in experimentation.

Findings

DER statistic effectively detects group representation deviations.

Methodology uncovers surprising insights about group inequality.

Practical application at LinkedIn demonstrates real-world utility.

Abstract

As companies adopt increasingly experimentation-driven cultures, it is crucial to develop methods for understanding any potential unintended consequences of those experiments. We might have specific questions about those consequences (did a change increase or decrease gender representation equality among content creators?); we might also wonder whether if we have not yet considered the right question (that is, we don't know what we don't know). Hence we address the problem of unintended consequences in experimentation from two perspectives: namely, pre-specified vs. data-driven selection, of dimensions of interest. For a specified dimension, we introduce a statistic to measure deviation from equal representation (DER statistic), give its asymptotic distribution, and evaluate finite-sample performance. We explain how to use this statistic to search across large-scale experimentation systems to alert us to any extreme unintended consequences on group representation. We complement this methodology by discussing a search for heterogeneous treatment effects along a set of dimensions with causal trees, modified slightly for practicalities in our ecosystem, and used here as a way to dive deeper into experiments flagged by the DER statistic alerts. We introduce a method for simulating data that closely mimics observed data at LinkedIn, and evaluate the performance of DER statistics in simulations. Last, we give a case study from LinkedIn, and show how these methodologies empowered us to discover surprising and important insights about group representation. Code for replication is available in an appendix.