Biased Minds Meet Biased AI: How Class Imbalance Shapes Appropriate Reliance and Interacts with Human Base Rate Neglect

TL;DR

This study investigates how class imbalance in AI systems influences human reliance and interacts with human biases like base rate neglect, revealing complex mutual reinforcement effects in decision-making.

Contribution

It provides empirical evidence on the interaction between AI class imbalance and human biases, highlighting the need for an interactionist approach in human-AI decision studies.

Findings

Class imbalance disrupts appropriate AI reliance

Mutually reinforcing effects between class imbalance and base rate neglect

Evidence of compound human-AI bias in decision-making

Abstract

Humans increasingly interact with artificial intelligence (AI) in decision-making. However, both AI and humans are prone to biases. While AI and human biases have been studied extensively in isolation, this paper examines their complex interaction. Specifically, we examined how class imbalance as an AI bias affects people's ability to appropriately rely on an AI-based decision-support system, and how it interacts with base rate neglect as a human bias. In a within-subject online study (N= 46), participants classified three diseases using an AI-based decision-support system trained on either a balanced or unbalanced dataset. We found that class imbalance disrupted participants' calibration of AI reliance. Moreover, we observed mutually reinforcing effects between class imbalance and base rate neglect, offering evidence of a compound human-AI bias. Based on these findings, we advocate for an interactionist perspective and further research into the mutually reinforcing effects of biases in human-AI interaction.