Improving Semantic Uncertainty Quantification in Language Model Question-Answering via Token-Level Temperature Scaling

TL;DR

This paper improves semantic uncertainty quantification in language model QA by introducing token-level temperature scaling, which enhances calibration and discrimination over existing heuristic methods.

Contribution

It demonstrates that optimizing a single scalar temperature for token-level calibration significantly improves semantic confidence measures in QA tasks.

Findings

Temperature scaling improves semantic calibration and discrimination.

Token-level temperature scaling outperforms heuristic baselines.

Enhanced uncertainty measures lead to better downstream entropy estimates.

Abstract

Calibration is central to reliable semantic uncertainty quantification, yet prior work has largely focused on discrimination, neglecting calibration. As calibration and discrimination capture distinct aspects of uncertainty, focusing on discrimination alone yields an incomplete picture. We address this gap by systematically evaluating both aspects across a broad set of confidence measures. We show that current approaches, particularly fixed-temperature heuristics, produce systematically miscalibrated and poorly discriminative semantic confidence distributions. We demonstrate that optimising a single scalar temperature, which, we argue, provides a suitable inductive bias, is a surprisingly simple yet effective solution. Our exhaustive evaluation confirms that temperature scaling consistently improves semantic calibration, discrimination, and downstream entropy, outperforming both heuristic baselines and more expressive token-level recalibration methods on question-answering tasks.