Misplaced Trust: Measuring the Interference of Machine Learning in Human Decision-Making

TL;DR

This study investigates how human trust in ML decision aids varies with expertise and information, revealing tendencies to overtrust and the influence of different informational cues across tasks.

Contribution

It provides empirical evidence on trust dynamics in ML-human interaction, highlighting factors that lead to misplaced trust regardless of user expertise.

Findings

People overtrust incorrect ML recommendations on familiar tasks.

All types of system information increased trust in recommendations.

Math and logic skills influence trust in ML systems.

Abstract

ML decision-aid systems are increasingly common on the web, but their successful integration relies on people trusting them appropriately: they should use the system to fill in gaps in their ability, but recognize signals that the system might be incorrect. We measured how people's trust in ML recommendations differs by expertise and with more system information through a task-based study of 175 adults. We used two tasks that are difficult for humans: comparing large crowd sizes and identifying similar-looking animals. Our results provide three key insights: (1) People trust incorrect ML recommendations for tasks that they perform correctly the majority of the time, even if they have high prior knowledge about ML or are given information indicating the system is not confident in its prediction; (2) Four different types of system information all increased people's trust in recommendations; and (3) Math and logic skills may be as important as ML for decision-makers working with ML recommendations.