Factors Influencing Perceived Fairness in Algorithmic Decision-Making: Algorithm Outcomes, Development Procedures, and Individual Differences

TL;DR

This study investigates how algorithm outcomes, development procedures, and individual differences influence perceptions of fairness in algorithmic decision-making, revealing outcome favorability bias and its moderation by user demographics.

Contribution

It provides new insights into the factors affecting perceived fairness, emphasizing the importance of outcome bias and individual differences in fairness evaluations.

Findings

People perceive algorithms as more fair when outcomes favor them.

Outcome favorability bias can overshadow the effects of algorithm bias.

Demographics and development procedures moderate fairness perceptions.

Abstract

Algorithmic decision-making systems are increasingly used throughout the public and private sectors to make important decisions or assist humans in making these decisions with real social consequences. While there has been substantial research in recent years to build fair decision-making algorithms, there has been less research seeking to understand the factors that affect people's perceptions of fairness in these systems, which we argue is also important for their broader acceptance. In this research, we conduct an online experiment to better understand perceptions of fairness, focusing on three sets of factors: algorithm outcomes, algorithm development and deployment procedures, and individual differences. We find that people rate the algorithm as more fair when the algorithm predicts in their favor, even surpassing the negative effects of describing algorithms that are very biased against particular demographic groups. We find that this effect is moderated by several variables, including participants' education level, gender, and several aspects of the development procedure. Our findings suggest that systems that evaluate algorithmic fairness through users' feedback must consider the possibility of outcome favorability bias.