Fairness through Experimentation: Inequality in A/B testing as an approach to responsible design

TL;DR

This paper introduces a novel experimentation-based approach to fairness in product design by measuring inequality impacts using the Atkinson index in A/B testing, promoting responsible and inclusive innovation.

Contribution

It presents a new method to assess fairness through inequality measurement in A/B testing, including causal inference and site-wide impact analysis, with real-world applications and scalable implementation.

Findings

Identified areas where product innovations foster fairness

Developed scalable tools for inequality measurement in experiments

Deployed method at scale with thousands of experiments

Abstract

As technology continues to advance, there is increasing concern about individuals being left behind. Many businesses are striving to adopt responsible design practices and avoid any unintended consequences of their products and services, ranging from privacy vulnerabilities to algorithmic bias. We propose a novel approach to fairness and inclusiveness based on experimentation. We use experimentation because we want to assess not only the intrinsic properties of products and algorithms but also their impact on people. We do this by introducing an inequality approach to A/B testing, leveraging the Atkinson index from the economics literature. We show how to perform causal inference over this inequality measure. We also introduce the concept of site-wide inequality impact, which captures the inclusiveness impact of targeting specific subpopulations for experiments, and show how to conduct statistical inference on this impact. We provide real examples from LinkedIn, as well as an open-source, highly scalable implementation of the computation of the Atkinson index and its variance in Spark/Scala. We also provide over a year's worth of learnings -- gathered by deploying our method at scale and analyzing thousands of experiments -- on which areas and which kinds of product innovations seem to inherently foster fairness through inclusiveness.