Beyond Accuracy: How AI Metacognitive Sensitivity improves AI-assisted Decision Making

TL;DR

This paper emphasizes the importance of AI metacognitive sensitivity, which accurately distinguishes correct from incorrect predictions, in improving human decision-making when combined with AI accuracy.

Contribution

It introduces a theoretical framework for assessing the combined effects of AI accuracy and metacognitive sensitivity on decision quality.

Findings

AI with higher metacognitive sensitivity can improve decision outcomes despite lower accuracy.

A behavioral experiment confirms the positive impact of AI metacognitive sensitivity on human decisions.

Evaluating AI assistance should include both accuracy and metacognitive sensitivity.

Abstract

In settings where human decision-making relies on AI input, both the predictive accuracy of the AI system and the reliability of its confidence estimates influence decision quality. We highlight the role of AI metacognitive sensitivity -- its ability to assign confidence scores that accurately distinguish correct from incorrect predictions -- and introduce a theoretical framework for assessing the joint impact of AI's predictive accuracy and metacognitive sensitivity in hybrid decision-making settings. Our analysis identifies conditions under which an AI with lower predictive accuracy but higher metacognitive sensitivity can enhance the overall accuracy of human decision making. Finally, a behavioral experiment confirms that greater AI metacognitive sensitivity improves human decision performance. Together, these findings underscore the importance of evaluating AI assistance not only by accuracy but also by metacognitive sensitivity, and of optimizing both to achieve superior decision outcomes.