AutoHall: Automated Factuality Hallucination Dataset Generation for Large Language Models

TL;DR

AutoHall introduces an automated method to generate model-specific hallucination datasets for large language models, enabling better detection and understanding of hallucinations without extensive manual annotation.

Contribution

The paper presents AutoHall, a novel automated approach to create hallucination datasets tailored to specific models, facilitating improved detection and analysis of hallucinations in LLMs.

Findings

Hallucination rates vary across different LLMs.

Self-contradiction based detection outperforms baselines.

Insights into factors influencing hallucinations.

Abstract

Large language models (LLMs) have gained broad applications across various domains but still struggle with hallucinations. Currently, hallucinations occur frequently in the generation of factual content and pose a great challenge to trustworthy LLMs. However, hallucination detection is hindered by the laborious and expensive manual annotation of hallucinatory content. Meanwhile, as different LLMs exhibit distinct types and rates of hallucination, the collection of hallucination datasets is inherently model-specific, which also increases the cost. To address this issue, this paper proposes a method called $\textbf{AutoHall}$ for $\underline{Auto}$matically constructing model-specific $\underline{Hall}$ucination datasets based on existing fact-checking datasets. The empirical results reveal variations in hallucination proportions and types among different models. Moreover, we introduce a zero-resource and black-box hallucination detection method based on self-contradiction to recognize the hallucination in our constructed dataset, achieving superior detection performance compared to baselines. Further analysis on our dataset provides insight into factors that may contribute to LLM hallucinations. Our codes and datasets are publicly available at https://github.com/zouyingcao/AutoHall.