It's only fair when I think it's fair: How Gender Bias Alignment Undermines Distributive Fairness in Human-AI Collaboration

TL;DR

This paper investigates how gender bias alignment between humans and AI affects fairness perceptions and reliance, revealing that formal fairness alone does not ensure effective collaboration when biases are misaligned.

Contribution

It introduces the concept of gender bias alignment in human-AI collaboration and demonstrates its impact on fairness perceptions and reliance through empirical study.

Findings

Gender bias alignment influences fairness perceptions.

Misaligned biases lead to overrides of AI recommendations.

Formal fairness does not guarantee effective collaboration.

Abstract

Human-AI collaboration is increasingly relevant in consequential areas where AI recommendations support human discretion. However, human-AI teams' effectiveness, capability, and fairness highly depend on human perceptions of AI. Positive fairness perceptions have been shown to foster trust and acceptance of AI recommendations. Yet, work on confirmation bias highlights that humans selectively adhere to AI recommendations that align with their expectations and beliefs -- despite not being necessarily correct or fair. This raises the question whether confirmation bias also transfers to the alignment of gender bias between human and AI decisions. In our study, we examine how gender bias alignment influences fairness perceptions and reliance. The results of a 2x2 between-subject study highlight the connection between gender bias alignment, fairness perceptions, and reliance, demonstrating that merely constructing a ``formally fair'' AI system is insufficient for optimal human-AI collaboration; ultimately, AI recommendations will likely be overridden if biases do not align.