Metacognition and Confidence Dynamics in Advice Taking from Generative AI

TL;DR

This study explores how confidence in oneself and GenAI influences advice-seeking and reliance, revealing that advice exposure boosts confidence and detailed responses often lack thorough checking, affecting decision quality.

Contribution

It introduces a novel paradigm to examine confidence dynamics in advice-taking from GenAI, demonstrating causal effects of advice exposure on confidence and reliance behaviors.

Findings

Advice-seeking increases confidence in GenAI and decreases confidence in self.

Advice exposure causally boosts retrospective confidence in both GenAI and self.

Participants often rely on verbose advice without thorough checking, missing key information.

Abstract

Generative Artificial Intelligence (GenAI) can aid humans in a wide range of tasks, but its effectiveness critically depends on users being able to evaluate the accuracy of GenAI outputs and their own expertise. Here we asked how confidence in self and GenAI contributes to decisions to seek and rely on advice from GenAI ('prospective confidence'), and how advice-taking in turn shapes this confidence ('retrospective confidence'). In a novel paradigm involving text generation, participants formulated plans for events, and could request advice from a GenAI (Study 1; N=200) or were randomly assigned to receive advice (Study 2; N=300), which they could rely on or ignore. Advice requests in Study 1 were related to higher prospective confidence in GenAI and lower confidence in self. Advice-seekers showed increased retrospective confidence in GenAI, while those who declined advice showed increased confidence in self. Random assignment in Study 2 revealed that advice exposure increases confidence in GenAI and in self, suggesting that GenAI advice-taking causally boosts retrospective confidence. These results were mirrored in advice reliance, operationalised as the textual similarity between GenAI advice and participants' responses, with reliance associated with increased retrospective confidence in both GenAI and self. Critically, participants who chose to obtain/rely on advice provided more detailed responses (likely due to the output's verbosity), but failed to check the output thoroughly, missing key information. These findings underscore a key role for confidence in interactions with GenAI, shaped by both prior beliefs about oneself and the reliability of AI, and context-dependent exposure to advice.