The Semantic Illusion: Certified Limits of Embedding-Based Hallucination Detection in RAG Systems

TL;DR

This paper investigates the limits of embedding-based hallucination detection in RAG systems, revealing a fundamental challenge where current methods fail on real hallucinations, but GPT-4 can effectively identify them through reasoning.

Contribution

The study introduces a conformal prediction framework for hallucination detection and uncovers the 'Semantic Illusion' phenomenon, showing the gap between surface-level detection and reasoning-based understanding.

Findings

Embedding methods achieve 95% coverage with 0% FPR on synthetic hallucinations.

Embedding methods fail catastrophically on real hallucinations, with 100% FPR.

GPT-4 achieves 7% FPR, demonstrating reasoning can overcome surface-level detection limits.

Abstract

Retrieval-Augmented Generation (RAG) systems remain susceptible to hallucinations despite grounding in retrieved evidence. While current detection methods leverage embedding similarity and natural language inference (NLI), their reliability in safety-critical settings remains unproven. We apply conformal prediction to RAG hallucination detection, transforming heuristic scores into decision sets with finite-sample coverage guarantees (1-alpha). Using calibration sets of n=600, we demonstrate a fundamental dichotomy: on synthetic hallucinations (Natural Questions), embedding methods achieve 95% coverage with 0% False Positive Rate (FPR). However, on real hallucinations from RLHF-aligned models (HaluEval), the same methods fail catastrophically, yielding 100% FPR at target coverage. We analyze this failure through the lens of distributional tails, showing that while NLI models achieve acceptable AUC (0.81), the "hardest" hallucinations are semantically indistinguishable from faithful responses, forcing conformal thresholds to reject nearly all valid outputs. Crucially, GPT-4 as a judge achieves 7% FPR (95% CI:[3.4%, 13.7%]) on the same data, proving the task is solvable via reasoning but opaque to surface-level semantics--a phenomenon we term the "Semantic Illusion."