A Multiple-Fill-in-the-Blank Exam Approach for Enhancing Zero-Resource Hallucination Detection in Large Language Models

TL;DR

This paper introduces a novel multiple-fill-in-the-blank exam approach to improve zero-resource hallucination detection in large language models, ensuring storyline consistency and outperforming existing methods.

Contribution

It proposes a new hallucination detection method that maintains storyline alignment by masking objects and repeatedly prompting LLMs, addressing the issue of story changes in regenerated texts.

Findings

Outperforms existing hallucination detection methods.

Achieves state-of-the-art performance in ensemble settings.

Effectively maintains storyline consistency during detection.

Abstract

Large language models (LLMs) often fabricate a hallucinatory text. Several methods have been developed to detect such text by semantically comparing it with the multiple versions probabilistically regenerated. However, a significant issue is that if the storyline of each regenerated text changes, the generated texts become incomparable, which worsen detection accuracy. In this paper, we propose a hallucination detection method that incorporates a multiple-fill-in-the-blank exam approach to address this storyline-changing issue. First, our method creates a multiple-fill-in-the-blank exam by masking multiple objects from the original text. Second, prompts an LLM to repeatedly answer this exam. This approach ensures that the storylines of the exam answers align with the original ones. Finally, quantifies the degree of hallucination for each original sentence by scoring the exam answers, considering the potential for \emph{hallucination snowballing} within the original text itself. Experimental results show that our method alone not only outperforms existing methods, but also achieves clearer state-of-the-art performance in the ensembles with existing methods.