Human-AI Interactions in Public Sector Decision-Making: "Automation Bias" and "Selective Adherence" to Algorithmic Advice

TL;DR

This paper investigates how public sector decision-makers may overrely on or selectively adhere to AI advice, potentially reinforcing biases and affecting vulnerable populations, based on three experimental studies.

Contribution

It identifies and analyzes automation bias and selective adherence in public sector AI use, highlighting their implications for fairness and decision quality.

Findings

Evidence of overreliance on AI advice despite warnings

Instances of selective adherence aligning with stereotypes

Potential negative impacts on disadvantaged citizens

Abstract

Artificial intelligence algorithms are increasingly adopted as decisional aides by public bodies, with the promise of overcoming biases of human decision-makers. At the same time, they may introduce new biases in the human-algorithm interaction. Drawing on psychology and public administration literatures, we investigate two key biases: overreliance on algorithmic advice even in the face of warning signals from other sources (automation bias), and selective adoption of algorithmic advice when this corresponds to stereotypes (selective adherence). We assess these via three experimental studies conducted in the NetherlandsWe discuss the implications of our findings for public sector decision making in the age of automation. Overall, our study speaks to potential negative effects of automation of the administrative state for already vulnerable and disadvantaged citizens.