Knowing What You Know Is Not Enough: Large Language Model Confidences Don't Align With Their Actions

TL;DR

This paper reveals that large language models' confidence estimates in static evaluations do not reliably predict their behavior in interactive, real-world scenarios, exposing a significant gap in current evaluation methods.

Contribution

It uncovers the action-belief gap in LLMs, demonstrating that static confidence calibration does not ensure rational actions in dynamic settings.

Findings

Models often bet against their high-confidence predictions in prediction markets.

Models fail to use information-seeking tools reliably when their confidence is low.

High confidence does not always lead to consistent or rational actions.

Abstract

Large language models (LLMs) are increasingly deployed in agentic and multi-turn workflows where they are tasked to perform actions of significant consequence. In order to deploy them reliably and manage risky outcomes in these settings, it is helpful to access model uncertainty estimates. However, confidence elicitation methods for LLMs are typically not evaluated directly in agentic settings; instead, they are evaluated on static datasets, such as Q&A benchmarks. In this work we investigate the relationship between confidence estimates elicited in static settings and the behavior of LLMs in interactive settings. We uncover a significant action-belief gap -- LLMs frequently take actions that contradict their elicited confidences. In a prediction market setting, we find that models often bet against their own high-confidence predictions; in a tool-use setting, models fail to reliably invoke information-seeking tools when their internal confidence is low; and in a user-challenge setting, models change their answers when they have high confidence in them, whilst sticking to answers they have low confidence in. Crucially, we show that static calibration is an insufficient predictor of consistency in the above dynamic settings, as stronger, better calibrated models are somtimes less consistent than their smaller and weaker open-source counterparts. Our results highlight a critical blind spot in current evaluation methodologies: ensuring that a model knows what it knows does not guarantee that it will act rationally on that knowledge.