Beyond AI advice -- independent aggregation boosts human-AI accuracy

TL;DR

This paper introduces the hybrid confirmation tree (HCT), an independent judgment aggregation method that outperforms traditional AI-as-advisor systems across multiple datasets by preserving human-AI judgment independence.

Contribution

The paper proposes the HCT approach, which maintains independence between human and AI judgments, improving decision accuracy and transparency over standard AI-advisor methods.

Findings

HCT outperforms AI-advisor in all tested datasets.

HCT performs better even when AI provides explanations.

People struggle to discriminate correct from incorrect AI advice.

Abstract

Artificial intelligence (AI) is broadly deployed as an advisor to human decision-makers: AI recommends a decision and a human accepts or rejects the advice. This approach, however, has several limitations: People frequently ignore accurate advice and rely too much on inaccurate advice, and their decision-making skills may deteriorate over time. Here, we compare the AI-as-advisor approach to the hybrid confirmation tree (HCT), an alternative strategy that preserves the independence of human and AI judgments. The HCT elicits a human judgment and an AI judgment independently of each other. If they agree, that decision is accepted. If not, a second human breaks the tie. For the comparison, we used 10 datasets from various domains, including medical diagnostics and misinformation discernment, and a subset of four datasets in which AI also explained its decision. The HCT outperformed the AI-as-advisor approach in all datasets. The HCT also performed better in almost all cases in which AI offered an explanation of its judgment. Using signal detection theory to interpret these results, we find that the HCT outperforms the AI-as-advisor approach because people cannot discriminate well enough between correct and incorrect AI advice. Overall, the HCT is a robust, accurate, and transparent alternative to the AI-as-advisor approach, offering a simple mechanism to tap into the wisdom of hybrid crowds.