HalluScan: A Systematic Benchmark for Detecting and Mitigating Hallucinations in Instruction-Following LLMs

TL;DR

HalluScan is a comprehensive benchmark framework designed to evaluate and improve hallucination detection and mitigation in instruction-following large language models across multiple configurations and domains.

Contribution

It introduces HalluScore, Adaptive Detection Routing, and a systematic error analysis, advancing the evaluation and reduction of hallucinations in LLMs.

Findings

HalluScore correlates with human judgments with r=0.41.

ADR reduces costs by 2.0x with minimal AUROC loss.

NLI Verification achieves AUROC of 0.88 in hallucination detection.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse natural language processing tasks, yet they remain susceptible to hallucinations -- generating content that is factually incorrect, unfaithful to provided context, or misaligned with user instructions. We present HalluScan, a comprehensive benchmark framework that systematically evaluates hallucination detection and mitigation across 72 configurations spanning 6 detection methods, 4 open-weight model families, and 3 diverse domains. We introduce three key contributions: (1) HalluScore, a novel composite metric that achieves a Pearson correlation of r = 0.41 with human expert judgments; (2) Adaptive Detection Routing (ADR), an intelligent routing algorithm achieving 2.0x cost reduction with only 0.1% AUROC degradation; and (3) systematic error cascade decomposition revealing substantial variation in hallucination error types across domains. Our experiments reveal that NLI Verification achieves the highest overall AUROC of 0.88, while RAV achieves the second-highest AUROC of 0.66.