Mind the Generation Process: Fine-Grained Confidence Estimation During LLM Generation

TL;DR

This paper introduces FineCE, a novel method for fine-grained, accurate confidence estimation during large language model generation, improving trustworthiness by predicting confidence scores throughout the text creation process.

Contribution

The paper presents a new confidence estimation approach, FineCE, with a training pipeline, a Backward Confidence Integration strategy, and position selection methods, advancing beyond coarse-grained existing techniques.

Findings

FineCE outperforms classical methods on benchmark datasets.

The approach provides continuous confidence scores during generation.

Experiments validate the effectiveness of the proposed strategies.

Abstract

While large language models (LLMs) have demonstrated remarkable performance across diverse tasks, they fundamentally lack self-awareness and frequently exhibit overconfidence, assigning high confidence scores to incorrect predictions. Accurate confidence estimation is therefore critical for enhancing the trustworthiness and reliability of LLM-generated outputs. However, existing approaches suffer from coarse-grained scoring mechanisms that fail to provide fine-grained, continuous confidence estimates throughout the generation process. To address these limitations, we introduce FineCE, a novel confidence estimation method that delivers accurate, fine-grained confidence scores during text generation. Specifically, we first develop a comprehensive pipeline for constructing training data that effectively captures the underlying probabilistic distribution of LLM responses, and then train a model to predict confidence scores for arbitrary text sequences in a supervised manner. Furthermore, we propose a Backward Confidence Integration (BCI) strategy that leverages information from the subsequent text to enhance confidence estimation for the current sequence during inference. We also introduce three strategies for identifying optimal positions to perform confidence estimation within the generation process. Extensive experiments on multiple benchmark datasets demonstrate that FineCE consistently outperforms existing classical confidence estimation methods. Our code and all baselines used in the paper are available on GitHub.