Training Towards Critical Use: Learning to Situate AI Predictions Relative to Human Knowledge

TL;DR

This paper introduces the concept of critical use, emphasizing human ability to contextualize AI predictions with unique knowledge, and demonstrates how targeted training can enhance this skill in complex social decision-making like child maltreatment screening.

Contribution

It proposes a new process-oriented framework for appropriate reliance called critical use and shows how training can improve human-AI decision alignment in real-world settings.

Findings

Participants learned to use qualitative case narratives alongside AI predictions.

Training led novices to adopt decision patterns similar to experienced workers.

The study highlights the importance of context-aware training for AI-assisted decisions.

Abstract

A growing body of research has explored how to support humans in making better use of AI-based decision support, including via training and onboarding. Existing research has focused on decision-making tasks where it is possible to evaluate "appropriate reliance" by comparing each decision against a ground truth label that cleanly maps to both the AI's predictive target and the human decision-maker's goals. However, this assumption does not hold in many real-world settings where AI tools are deployed today (e.g., social work, criminal justice, and healthcare). In this paper, we introduce a process-oriented notion of appropriate reliance called critical use that centers the human's ability to situate AI predictions against knowledge that is uniquely available to them but unavailable to the AI model. To explore how training can support critical use, we conduct a randomized online experiment in a complex social decision-making setting: child maltreatment screening. We find that, by providing participants with accelerated, low-stakes opportunities to practice AI-assisted decision-making in this setting, novices came to exhibit patterns of disagreement with AI that resemble those of experienced workers. A qualitative examination of participants' explanations for their AI-assisted decisions revealed that they drew upon qualitative case narratives, to which the AI model did not have access, to learn when (not) to rely on AI predictions. Our findings open new questions for the study and design of training for real-world AI-assisted decision-making.