The HalluRAG Dataset: Detecting Closed-Domain Hallucinations in RAG Applications Using an LLM's Internal States

TL;DR

This paper introduces HalluRAG, a dataset for detecting recent, ungrounded hallucinations in large language models at the sentence level, using internal states, with classifiers achieving up to 75% accuracy.

Contribution

The study presents HalluRAG, a novel dataset for training hallucination classifiers based on internal LLM states, and evaluates their effectiveness across different models and configurations.

Findings

Classifiers trained on HalluRAG reach up to 75% accuracy.

Internal states encode distinguishable signals for hallucinated vs. factual sentences.

Answerable and unanswerable prompts are encoded differently, improving detection.

Abstract

Detecting hallucinations in large language models (LLMs) is critical for enhancing their reliability and trustworthiness. Most research focuses on hallucinations as deviations from information seen during training. However, the opaque nature of an LLM's parametric knowledge complicates the understanding of why generated texts appear ungrounded: The LLM might not have picked up the necessary knowledge from large and often inaccessible datasets, or the information might have been changed or contradicted during further training. Our focus is on hallucinations involving information not used in training, which we determine by using recency to ensure the information emerged after a cut-off date. This study investigates these hallucinations by detecting them at sentence level using different internal states of various LLMs. We present HalluRAG, a dataset designed to train classifiers on these hallucinations. Depending on the model and quantization, MLPs trained on HalluRAG detect hallucinations with test accuracies ranging up to 75 %, with Mistral-7B-Instruct-v0.1 achieving the highest test accuracies. Our results show that IAVs detect hallucinations as effectively as CEVs and reveal that answerable and unanswerable prompts are encoded differently as separate classifiers for these categories improved accuracy. However, HalluRAG showed some limited generalizability, advocating for more diversity in datasets on hallucinations.