Uncertainty Profiles for LLMs: Uncertainty Source Decomposition and Adaptive Model-Metric Selection

TL;DR

This paper introduces a framework to decompose and analyze uncertainty in large language models, enabling better model and metric selection for more reliable outputs.

Contribution

It presents a novel systematic decomposition of LLM uncertainty into four sources and a method for task-specific model and metric selection based on uncertainty characteristics.

Findings

Uncertainty metrics vary systematically across tasks and models.

The proposed selection strategy improves reliability over baseline methods.

Decomposition aids in understanding and managing LLM hallucinations.

Abstract

Large language models (LLMs) often generate fluent but factually incorrect outputs, known as hallucinations, which undermine their reliability in real-world applications. While uncertainty estimation has emerged as a promising strategy for detecting such errors, current metrics offer limited interpretability and lack clarity about the types of uncertainty they capture. In this paper, we present a systematic framework for decomposing LLM uncertainty into four distinct sources, inspired by previous research. We develop a source-specific estimation pipeline to quantify these uncertainty types and evaluate how existing metrics relate to each source across tasks and models. Our results show that metrics, task, and model exhibit systematic variation in uncertainty characteristic. Building on this, we propose a method for task specific metric/model selection guided by the alignment or divergence between their uncertainty characteristics and that of a given task. Our experiments across datasets and models demonstrate that our uncertainty-aware selection strategy consistently outperforms baseline strategies, helping us select appropriate models or uncertainty metrics, and contributing to more reliable and efficient deployment in uncertainty estimation.