Visualizing and Benchmarking LLM Factual Hallucination Tendencies via Internal State Analysis and Clustering

TL;DR

This paper introduces FalseCite, a dataset for benchmarking LLM hallucinations caused by false citations, and analyzes internal model states to understand and visualize hallucination tendencies.

Contribution

It presents a new dataset, FalseCite, and demonstrates how internal state analysis can reveal patterns in hallucination behavior across different LLMs.

Findings

Hallucination activity increases with false citations, especially in GPT-4o-mini.

Internal hidden states form a distinct horn-like shape during hallucinations.

FalseCite effectively benchmarks and studies LLM hallucination tendencies.

Abstract

Large Language Models (LLMs) often hallucinate, generating nonsensical or false information that can be especially harmful in sensitive fields such as medicine or law. To study this phenomenon systematically, we introduce FalseCite, a curated dataset designed to capture and benchmark hallucinated responses induced by misleading or fabricated citations. Running GPT-4o-mini, Falcon-7B, and Mistral 7-B through FalseCite, we observed a noticeable increase in hallucination activity for false claims with deceptive citations, especially in GPT-4o-mini. Using the responses from FalseCite, we can also analyze the internal states of hallucinating models, visualizing and clustering the hidden state vectors. From this analysis, we noticed that the hidden state vectors, regardless of hallucination or non-hallucination, tend to trace out a distinct horn-like shape. Our work underscores FalseCite's potential as a foundation for evaluating and mitigating hallucinations in future LLM research.