Double-Calibration: Towards Reliable LLMs via Calibrating Knowledge and Reasoning Confidence

TL;DR

DoublyCal is a framework that enhances the reliability of Large Language Models by calibrating both knowledge evidence and reasoning confidence, leading to more accurate and trustworthy predictions.

Contribution

It introduces a novel double-calibration principle and a lightweight proxy model to improve LLMs' factual accuracy and confidence calibration.

Findings

Significantly improves accuracy on knowledge-intensive benchmarks.

Enhances confidence calibration of black-box LLMs.

Maintains low token cost during inference.

Abstract

Reliable reasoning in Large Language Models (LLMs) is challenged by their propensity for hallucination. While augmenting LLMs with Knowledge Graphs (KGs) improves factual accuracy, existing KG-augmented methods fail to quantify epistemic uncertainty in both the retrieved evidence and LLMs' reasoning. To bridge this gap, we introduce DoublyCal, a framework built on a novel double-calibration principle. DoublyCal employs a lightweight proxy model to first generate KG evidence alongside a calibrated evidence confidence. This calibrated supporting evidence then guides a black-box LLM, yielding final predictions that are not only more accurate but also well-calibrated, with confidence scores traceable to the uncertainty of the supporting evidence. Experiments on knowledge-intensive benchmarks show that DoublyCal significantly improves both the accuracy and confidence calibration of black-box LLMs while maintaining low token cost.