An Outcome Test of Discrimination for Ranked Lists

TL;DR

This paper develops a statistical test for discrimination in ranked lists, applicable to online search results and feeds, based on extending outcome tests to ordinal rankings and validating with LinkedIn data.

Contribution

It introduces a novel system of moment inequalities for testing discrimination in ranked lists, the only testable implications when only outcomes and group membership are observed.

Findings

The moment inequalities can be statistically tested for discrimination.

The approach is validated using real-world LinkedIn data.

Non-discrimination implies specific inequalities that can be empirically tested.

Abstract

This paper extends Becker (1957)'s outcome test of discrimination to settings where a (human or algorithmic) decision-maker produces a ranked list of candidates. Ranked lists are particularly relevant in the context of online platforms that produce search results or feeds, and also arise when human decisionmakers express ordinal preferences over a list of candidates. We show that non-discrimination implies a system of moment inequalities, which intuitively impose that one cannot permute the position of a lower-ranked candidate from one group with a higher-ranked candidate from a second group and systematically improve the objective. Moreover, we show that that these moment inequalities are the only testable implications of non-discrimination when the auditor observes only outcomes and group membership by rank. We show how to statistically test the implied inequalities, and validate our approach in an application using data from LinkedIn.