UNCERTAINTY-LINE: Length-Invariant Estimation of Uncertainty for Large Language Models

TL;DR

The paper introduces UNCERTAINTY-LINE, a simple post-hoc debiasing method that produces length-invariant uncertainty estimates for large language models, improving reliability across various tasks.

Contribution

It proposes a novel length-invariant uncertainty estimation technique that corrects biases in existing methods, applicable across different models and tasks.

Findings

UNCERTAINTY-LINE improves uncertainty estimates in translation, summarization, and QA.

The method is model-agnostic and post-hoc, requiring no retraining.

It outperforms existing length-normalized uncertainty measures.

Abstract

Large Language Models (LLMs) have become indispensable tools across various applications, making it more important than ever to ensure the quality and the trustworthiness of their outputs. This has led to growing interest in uncertainty quantification (UQ) methods for assessing the reliability of LLM outputs. Many existing UQ techniques rely on token probabilities, which inadvertently introduces a bias with respect to the length of the output. While some methods attempt to account for this, we demonstrate that such biases persist even in length-normalized approaches. To address the problem, here we propose UNCERTAINTY-LINE: (Length-INvariant Estimation), a simple debiasing procedure that regresses uncertainty scores on output length and uses the residuals as corrected, length-invariant estimates. Our method is post-hoc, model-agnostic, and applicable to a range of UQ measures. Through extensive evaluation on machine translation, summarization, and question-answering tasks, we demonstrate that UNCERTAINTY-LINE: consistently improves over even nominally length-normalized UQ methods uncertainty estimates across multiple metrics and models.