Verify when Uncertain: Beyond Self-Consistency in Black Box Hallucination Detection

TL;DR

This paper investigates hallucination detection in large language models, showing limitations of self-consistency methods and proposing a cross-model, cost-effective detection approach with theoretical insights.

Contribution

It introduces a cross-model consistency checking method and a two-stage detection algorithm that reduces costs while improving detection accuracy.

Findings

Cross-model checking outperforms self-consistency alone.

The two-stage method reduces computational costs significantly.

Theoretical analysis provides new geometric insights into detection methods.

Abstract

Large Language Models (LLMs) suffer from hallucination problems, which hinder their reliability in sensitive applications. In the black-box setting, several self-consistency-based techniques have been proposed for hallucination detection. We empirically study these techniques and show that they achieve performance close to that of a supervised (still black-box) oracle, suggesting little room for improvement within this paradigm. To address this limitation, we explore cross-model consistency checking between the target model and an additional verifier LLM. With this extra information, we observe improved oracle performance compared to purely self-consistency-based methods. We then propose a budget-friendly, two-stage detection algorithm that calls the verifier model only for a subset of cases. It dynamically switches between self-consistency and cross-consistency based on an uncertainty interval of the self-consistency classifier. We provide a geometric interpretation of consistency-based hallucination detection methods through the lens of kernel mean embeddings, offering deeper theoretical insights. Extensive experiments show that this approach maintains high detection performance while significantly reducing computational cost.