# Confidence-weighted integration of human and machine judgments for superior decision-making

## TL;DR

This paper shows that combining human and AI judgments using confidence weights improves decision-making, even when AI outperforms humans.

## Contribution

A scalable, confidence-weighted method for integrating human and machine judgments in decision-making.

## Key findings

- Human-AI teams outperformed individual AI systems in forecasting tasks.
- Adding a human to a team of AI systems consistently improved performance.
- Confidence-weighted integration is effective across different domains like image classification and neuroscience.

## Abstract

Large language models (LLMs) can surpass humans in certain forecasting tasks. What role does this leave for humans in the overall decision process? One possibility is that humans, despite performing worse than LLMs, can still add value when teamed with them. A human and machine team can surpass each individual teammate when team members’ confidence is well calibrated and team members diverge in which tasks they find difficult (i.e., calibration and diversity are needed). We simplified and extended a Bayesian approach to combining judgments using a logistic regression framework that integrates confidence-weighted judgments for any number of team members. Using this straightforward method, we demonstrated its effectiveness in both image classification and neuroscience forecasting tasks. Combining human judgments with one or more machines consistently improved overall team performance. Our hope is that this simple and effective strategy for integrating the judgments of humans and machines will lead to productive collaborations.

•Requirements for effective human-AI teaming, even when AI exceeds human capabilities•Scalable approach combining confidence-weighted judgments from humans and AI•Human-AI teaming surpassed individual AI systems in two different forecasting tasks•Adding a human to a team with one or more AI systems improved team performance

Requirements for effective human-AI teaming, even when AI exceeds human capabilities

Scalable approach combining confidence-weighted judgments from humans and AI

Human-AI teaming surpassed individual AI systems in two different forecasting tasks

Adding a human to a team with one or more AI systems improved team performance

Artificial intelligence (AI) systems, such as large language models (LLMs), have emerged as powerful tools in various domains. Recent studies have shown that LLMs can surpass humans in certain tasks, such as predicting the outcomes of neuroscience studies. This raises a critical question: as AI systems achieve superhuman performance in specific domains, will they displace human judgment in critical decision-making processes? Our study shows that humans still have a lot to offer. Because humans tend to make different mistakes than machines and can express how confident they are in their decisions, human judgments can be combined with those of AI systems to form teams that are more effective than teams consisting of machines alone. At least in the near term, human judgments offer a valuable and complementary signal that can increase decision-making performance in human-machine teams or ensembles.

When AI surpasses human performance, what can humans offer? We demonstrate that the performance of teams increases by integrating human judgments with those of machines. Integration is achieved by a straightforward regression approach that combines team members' confidence-weighted judgments.