More Isn't Always Better: Balancing Decision Accuracy and Conformity Pressures in Multi-AI Advice

TL;DR

This study investigates how multiple AI advice sources influence human decision-making, revealing that small panels can improve accuracy but larger panels and consensus levels can cause overreliance or confusion, guiding better multi-AI presentation design.

Contribution

It systematically examines the effects of panel size, consensus, and presentation style on human reliance and accuracy in multi-AI advice, providing practical design insights.

Findings

Small AI panels improve accuracy over single AI.

Large panels do not yield additional benefits.

High consensus increases overreliance; dissent reduces conformity pressure.

Abstract

Just as people improve decision-making by consulting diverse human advisors, they can now also consult with multiple AI systems. Prior work on group decision-making shows that advice aggregation creates pressure to conform, leading to overreliance. However, the conditions under which multi-AI consultation improves or undermines human decision-making remain unclear. We conducted experiments with three tasks in which participants received advice from panels of AIs. We varied panel size, within-panel consensus, and the human-likeness of presentation. Accuracy improved for small panels relative to a single AI; larger panels yielded no gains. The level of within-panel consensus affected participants' reliance on AI advice: High consensus fostered overreliance; a single dissent reduced pressure to conform; wide disagreement created confusion and undermined appropriate reliance. Human-like presentations increased perceived usefulness and agency in certain tasks, without raising conformity pressure. These findings yield design implications for presenting multi-AI advice that preserve accuracy while mitigating conformity.