The FairCeptron: A Framework for Measuring Human Perceptions of Algorithmic Fairness

TL;DR

The paper introduces the FairCeptron framework for studying and comparing human perceptions of fairness in algorithmic decision making, emphasizing the importance of human-centered fairness assessments.

Contribution

It presents a novel framework that integrates scenario generation, perception elicitation, and analysis to study human fairness perceptions and compare them with algorithmic fairness measures.

Findings

Applied to a university admission scenario involving minorities.

Framework implementation is openly available for adaptation.

Highlights the variability of human fairness perceptions across sociodemographics.

Abstract

Measures of algorithmic fairness often do not account for human perceptions of fairness that can substantially vary between different sociodemographics and stakeholders. The FairCeptron framework is an approach for studying perceptions of fairness in algorithmic decision making such as in ranking or classification. It supports (i) studying human perceptions of fairness and (ii) comparing these human perceptions with measures of algorithmic fairness. The framework includes fairness scenario generation, fairness perception elicitation and fairness perception analysis. We demonstrate the FairCeptron framework by applying it to a hypothetical university admission context where we collect human perceptions of fairness in the presence of minorities. An implementation of the FairCeptron framework is openly available, and it can easily be adapted to study perceptions of algorithmic fairness in other application contexts. We hope our work paves the way towards elevating the role of studies of human fairness perceptions in the process of designing algorithmic decision making systems.