Lowest Span Confidence: A Zero-Shot Metric for Efficient and Black-Box Hallucination Detection in LLMs

TL;DR

This paper introduces Lowest Span Confidence (LSC), a zero-shot, resource-efficient metric that detects hallucinations in large language models by analyzing local span likelihoods, outperforming existing methods across various benchmarks.

Contribution

The paper proposes LSC, a novel zero-shot hallucination detection metric that requires only a single model pass and effectively captures local uncertainty patterns, improving detection accuracy.

Findings

LSC outperforms existing zero-shot baselines in multiple benchmarks.

LSC is robust under resource-constrained conditions.

LSC effectively detects factual inconsistencies in LLM outputs.

Abstract

Hallucinations in Large Language Models (LLMs), i.e., the tendency to generate plausible but non-factual content, pose a significant challenge for their reliable deployment in high-stakes environments. However, existing hallucination detection methods generally operate under unrealistic assumptions, i.e., either requiring expensive intensive sampling strategies for consistency checks or white-box LLM states, which are unavailable or inefficient in common API-based scenarios. To this end, we propose a novel efficient zero-shot metric called Lowest Span Confidence (LSC) for hallucination detection under minimal resource assumptions, only requiring a single forward with output probabilities. Concretely, LSC evaluates the joint likelihood of semantically coherent spans via a sliding window mechanism. By identifying regions of lowest marginal confidence across variable-length n-grams, LSC could well capture local uncertainty patterns strongly correlated with factual inconsistency. Importantly, LSC can mitigate the dilution effect of perplexity and the noise sensitivity of minimum token probability, offering a more robust estimate of factual uncertainty. Extensive experiments across multiple state-of-the-art (SOTA) LLMs and diverse benchmarks show that LSC consistently outperforms existing zero-shot baselines, delivering strong detection performance even under resource-constrained conditions.