Influential Training Data Retrieval for Explaining Verbalized Confidence of LLMs

TL;DR

This paper introduces TracVC, a method to trace LLMs' verbalized confidence back to training data, revealing that models often rely on superficial cues rather than content grounding, which impacts trustworthiness.

Contribution

The paper presents TracVC, a novel influence estimation technique for analyzing the sources of LLMs' confidence expressions, and introduces content groundness as a new evaluation metric.

Findings

OLMo2-13B often relies on confidence data unrelated to queries.

Models tend to mimic superficial linguistic confidence cues.

Current training regimes may not ensure confidence is content-justified.

Abstract

Large language models (LLMs) can increase users' perceived trust by verbalizing confidence in their outputs. However, prior work has shown that LLMs are often overconfident, making their stated confidence unreliable since it does not consistently align with factual accuracy. To better understand the sources of this verbalized confidence, we introduce TracVC (\textbf{Trac}ing \textbf{V}erbalized \textbf{C}onfidence), a method that builds on information retrieval and influence estimation to trace generated confidence expressions back to the training data. We evaluate TracVC on OLMo and Llama models in a question answering setting, proposing a new metric, content groundness, which measures the extent to which an LLM grounds its confidence in content-related training examples (relevant to the question and answer) versus in generic examples of confidence verbalization. Our analysis reveals that OLMo2-13B is frequently influenced by confidence-related data that is lexically unrelated to the query, suggesting that it may mimic superficial linguistic expressions of certainty rather than rely on genuine content grounding. These findings point to a fundamental limitation in current training regimes: LLMs may learn how to sound confident without learning when confidence is justified. Our analysis provides a foundation for improving LLMs' trustworthiness in expressing more reliable confidence.