Retrieval-Augmented Linguistic Calibration

TL;DR

This paper introduces a distributional framework for linguistic confidence calibration, proposes a new metric called Faithfulness Divergence, and presents RALC, a retrieval-augmented rewriting pipeline that improves faithfulness and calibration in language models.

Contribution

It develops a novel distributional approach to linguistic confidence, introduces FD as a new evaluation metric, and presents RALC, a post-hoc calibration method that enhances model faithfulness and calibration.

Findings

RALC improves faithfulness by up to 66%

RALC enhances calibration accuracy by up to 58%

The distributional framework captures interpretation variability effectively

Abstract

Linguistic cues such as "I believe" and "probably" offer an intuitive interface for communicating confidence, yet a generalisable, principled calibration framework for linguistic confidence expressions remains underexplored. In particular, co-occurring linguistic cues, contextual variation, and subjective audience interpretation pose unique challenges. We therefore model linguistic confidence as a distribution over plausible perceived probability values that a statement is correct, capturing interpretation variability that scalar representations discard. Within this distributional framework, we introduce faithfulness as a complementary evaluation dimension and present Faithfulness Divergence (FD), an information-theoretic metric quantifying the surprise induced in audience beliefs upon truth revelation. Building on these foundations, we present Retrieval-Augmented Linguistic Calibration (RALC), a lightweight post-hoc pipeline that propagates calibrated confidence signals back into natural language via retrieval-augmented rewriting. Across three QA benchmarks and five LLM families, RALC improves in-domain faithfulness and calibration up to 66% and 58%, respectively, outperforming black-box and grey-box calibration baselines.