Rethinking Hallucinations: Correctness, Consistency, and Prompt Multiplicity

TL;DR

This paper introduces prompt multiplicity to evaluate the consistency of large language models, revealing significant inconsistencies in hallucination detection and mitigation, and highlighting the need for more comprehensive evaluation methods.

Contribution

It proposes prompt multiplicity as a new framework for assessing consistency in LLM hallucination evaluation, exposing limitations in current detection and mitigation techniques.

Findings

Over 50% inconsistency in benchmarks like Med-HALT

Detection techniques focus on consistency, not correctness

Mitigation methods like RAG can increase inconsistencies

Abstract

Large language models (LLMs) are known to "hallucinate" by generating false or misleading outputs. Hallucinations pose various harms, from erosion of trust to widespread misinformation. Existing hallucination evaluation, however, focuses only on correctness and often overlooks consistency, necessary to distinguish and address these harms. To bridge this gap, we introduce prompt multiplicity, a framework for quantifying consistency in LLM evaluations. Our analysis reveals significant multiplicity (over 50% inconsistency in benchmarks like Med-HALT), suggesting that hallucination-related harms have been severely misunderstood. Furthermore, we study the role of consistency in hallucination detection and mitigation. We find that: (a) detection techniques detect consistency, not correctness, and (b) mitigation techniques like RAG, while beneficial, can introduce additional inconsistencies. By integrating prompt multiplicity into hallucination evaluation, we provide an improved framework of potential harms and uncover critical limitations in current detection and mitigation strategies.