HIDE and Seek: Detecting Hallucinations in Language Models via Decoupled Representations

TL;DR

This paper introduces HIDE, a training-free, single-pass method that detects hallucinations in language models by analyzing the statistical decoupling of internal representations, improving detection accuracy and efficiency.

Contribution

HIDE leverages internal representation decoupling and HSIC to detect hallucinations in a single pass without additional training, outperforming existing methods in accuracy and computational efficiency.

Findings

HIDE outperforms other single-pass methods in AUC-ROC by ~29%.

HIDE achieves comparable or better results than multi-pass methods, with ~51% less computation.

Extensive experiments on diverse datasets and models validate HIDE's effectiveness.

Abstract

Contemporary Language Models (LMs), while impressively fluent, often generate content that is factually incorrect or unfaithful to the input context - a critical issue commonly referred to as 'hallucination'. This tendency of LMs to generate hallucinated content undermines their reliability, especially because these fabrications are often highly convincing and therefore difficult to detect. While several existing methods attempt to detect hallucinations, most rely on analyzing multiple generations per input, leading to increased computational cost and latency. To address this, we propose a single-pass, training-free approach for effective Hallucination detectIon via Decoupled rEpresentations (HIDE). Our approach leverages the hypothesis that hallucinations result from a statistical decoupling between an LM's internal representations of input context and its generated output. We quantify this decoupling using the Hilbert-Schmidt Independence Criterion (HSIC) applied to hidden-state representations extracted while generating the output sequence. We conduct extensive experiments on four diverse question answering datasets, evaluating both faithfulness and factuality hallucinations across six open-source LMs of varying scales and properties. Our results demonstrate that HIDE outperforms other single-pass methods in almost all settings, achieving an average relative improvement of ~29% in AUC-ROC over the best-performing single-pass strategy across various models and datasets. Additionally, HIDE shows competitive and often superior performance with multi-pass state-of-the-art methods, obtaining an average relative improvement of ~3% in AUC-ROC while consuming ~51% less computation time. Our findings highlight the effectiveness of exploiting internal representation decoupling in LMs for efficient and practical hallucination detection.