RePPL: Recalibrating Perplexity by Uncertainty in Semantic Propagation and Language Generation for Explainable QA Hallucination Detection

TL;DR

RePPL introduces a novel method to improve hallucination detection in large language models by recalibrating uncertainty measurements based on semantic propagation and language generation, providing explainable token-level insights.

Contribution

It proposes RePPL, a new approach that recalibrates uncertainty scores for better hallucination detection and explanation in LLMs, addressing limitations of previous uncertainty-based methods.

Findings

Achieves an average AUC of 0.833 across QA datasets.

Provides token-level uncertainty scores as explanations.

Outperforms previous methods in hallucination detection.

Abstract

Large Language Models (LLMs) have become powerful, but hallucinations remain a vital obstacle to their trustworthy use. Previous works improved the capability of hallucination detection by measuring uncertainty. But they can not explain the provenance behind why hallucinations occur, particularly in identifying which part of the inputs tends to trigger hallucinations. Recent works on the prompt attack indicate that uncertainty exists in semantic propagation, where attention mechanisms gradually fuse local token information into high-level semantics across layers. Meanwhile, uncertainty also emerges in language generation, due to its probability-based selection of high-level semantics for sampled generations. Based on that, we propose RePPL to recalibrate uncertainty measurement by these two aspects, which dispatches explainable uncertainty scores to each token and aggregates in Perplexity-style Log-Average form as a total score. Experiments show that it achieves the best comprehensive detection performance across various QA datasets on advanced models (average AUC of 0.833), and it is capable of producing token-level uncertainty scores as explanations of hallucination.