Measuring Perceptions of Fairness in AI Systems: The Effects of Infra-marginality

TL;DR

This study investigates how human perceptions of fairness in AI are influenced by data distribution differences, revealing that fairness judgments depend on outcome equality and beliefs about disparities, which impacts fairness metrics.

Contribution

It provides empirical evidence that fairness perceptions are context-dependent and influenced by data distribution, challenging traditional fairness definitions.

Findings

Participants preferred outcome equality when data was balanced or unavailable.

Perceptions of fairness aligned with data imbalances when differences reflected real disparities.

Fairness judgments are influenced by beliefs about the causes of disparities.

Abstract

Differences in data distributions between demographic groups, known as the problem of infra-marginality, complicate how people evaluate fairness in machine learning models. We present a user study with 85 participants in a hypothetical medical decision-making scenario to examine two treatments: group-specific model performance and training data availability. Our results show that participants did not equate fairness with simple statistical parity. When group-specific performances were equal or unavailable, participants preferred models that produced equal outcomes; when performances differed, especially in ways consistent with data imbalances, they judged models that preserved those differences as more fair. These findings highlight that fairness judgments are shaped not only by outcomes, but also by beliefs about the causes of disparities. We discuss implications for popular group fairness definitions and system design, arguing that accounting for distributional context is critical to aligning algorithmic fairness metrics with human expectations in real-world applications.