To Err Is AI! Debugging as an Intervention to Facilitate Appropriate Reliance on AI Systems

TL;DR

This study investigates whether debugging AI systems can improve human users' ability to appropriately rely on AI advice, but finds that debugging may decrease reliance due to exposing AI weaknesses early.

Contribution

The paper introduces the idea of using debugging as an intervention to calibrate human trust in AI, and empirically tests its effectiveness in a human-AI collaboration context.

Findings

Debugging intervention did not increase appropriate reliance on AI.

Participants' reliance decreased after debugging, possibly due to early exposure to AI weaknesses.

User confidence and trustworthiness estimates vary with AI performance levels.

Abstract

Powerful predictive AI systems have demonstrated great potential in augmenting human decision making. Recent empirical work has argued that the vision for optimal human-AI collaboration requires 'appropriate reliance' of humans on AI systems. However, accurately estimating the trustworthiness of AI advice at the instance level is quite challenging, especially in the absence of performance feedback pertaining to the AI system. In practice, the performance disparity of machine learning models on out-of-distribution data makes the dataset-specific performance feedback unreliable in human-AI collaboration. Inspired by existing literature on critical thinking and a critical mindset, we propose the use of debugging an AI system as an intervention to foster appropriate reliance. In this paper, we explore whether a critical evaluation of AI performance within a debugging setting can better calibrate users' assessment of an AI system and lead to more appropriate reliance. Through a quantitative empirical study (N = 234), we found that our proposed debugging intervention does not work as expected in facilitating appropriate reliance. Instead, we observe a decrease in reliance on the AI system after the intervention -- potentially resulting from an early exposure to the AI system's weakness. We explore the dynamics of user confidence and user estimation of AI trustworthiness across groups with different performance levels to help explain how inappropriate reliance patterns occur. Our findings have important implications for designing effective interventions to facilitate appropriate reliance and better human-AI collaboration.