What are Models Thinking about? Understanding Large Language Model Hallucinations "Psychology" through Model Inner State Analysis

TL;DR

This paper investigates the internal states of large language models during inference to understand and detect hallucinations without relying on external sources, providing insights into model behavior and improving detection methods.

Contribution

It introduces a systematic analysis of LLM internal states during different inference stages and evaluates their effectiveness in hallucination detection, enhancing interpretability.

Findings

Internal states reveal key features associated with hallucinations.

Analyzing stages helps understand why hallucinations occur.

Internal state-based detection has advantages and limitations.

Abstract

Large language model (LLM) systems suffer from the models' unstable ability to generate valid and factual content, resulting in hallucination generation. Current hallucination detection methods heavily rely on out-of-model information sources, such as RAG to assist the detection, thus bringing heavy additional latency. Recently, internal states of LLMs' inference have been widely used in numerous research works, such as prompt injection detection, etc. Considering the interpretability of LLM internal states and the fact that they do not require external information sources, we introduce such states into LLM hallucination detection. In this paper, we systematically analyze different internal states' revealing features during inference forward and comprehensively evaluate their ability in hallucination detection. Specifically, we cut the forward process of a large language model into three stages: understanding, query, generation, and extracting the internal state from these stages. By analyzing these states, we provide a deep understanding of why the hallucinated content is generated and what happened in the internal state of the models. Then, we introduce these internal states into hallucination detection and conduct comprehensive experiments to discuss the advantages and limitations.