Does More Advice Help? The Effects of Second Opinions in AI-Assisted Decision Making

TL;DR

This study investigates how providing second opinions in AI-assisted decision-making influences user reliance, finding that controlled solicitation can improve collaboration by reducing over-reliance without increasing under-reliance.

Contribution

It demonstrates that allowing decision-makers to actively seek second opinions can mitigate over-reliance on AI, offering new insights into improving human-AI collaboration strategies.

Findings

Simultaneous presentation of AI and second opinions reduces over-reliance but increases under-reliance.

Active solicitation of second opinions can mitigate over-reliance without causing under-reliance.

Control over when to seek second opinions influences decision-maker reliance behavior.

Abstract

AI assistance in decision-making has become popular, yet people's inappropriate reliance on AI often leads to unsatisfactory human-AI collaboration performance. In this paper, through three pre-registered, randomized human subject experiments, we explore whether and how the provision of {second opinions} may affect decision-makers' behavior and performance in AI-assisted decision-making. We find that if both the AI model's decision recommendation and a second opinion are always presented together, decision-makers reduce their over-reliance on AI while increase their under-reliance on AI, regardless whether the second opinion is generated by a peer or another AI model. However, if decision-makers have the control to decide when to solicit a peer's second opinion, we find that their active solicitations of second opinions have the potential to mitigate over-reliance on AI without inducing increased under-reliance in some cases. We conclude by discussing the implications of our findings for promoting effective human-AI collaborations in decision-making.