The System Hallucination Scale (SHS): A Minimal yet Effective Human-Centered Instrument for Evaluating Hallucination-Related Behavior in Large Language Models

TL;DR

The paper presents the System Hallucination Scale (SHS), a human-centered, lightweight tool for evaluating hallucination-related behaviors in large language models through user interaction, emphasizing interpretability and practicality.

Contribution

It introduces SHS as a novel, psychometrically validated instrument inspired by usability scales, specifically designed for assessing hallucination phenomena in LLMs from a user perspective.

Findings

High internal consistency (Cronbach's alpha = 0.87)

Significant correlations between dimensions (p < 0.001)

Demonstrated effectiveness in real-world user studies

Abstract

We introduce the System Hallucination Scale (SHS), a lightweight and human-centered measurement instrument for assessing hallucination-related behavior in large language models (LLMs). Inspired by established psychometric tools such as the System Usability Scale (SUS) and the System Causability Scale (SCS), SHS enables rapid, interpretable, and domain-agnostic evaluation of factual unreliability, incoherence, misleading presentation, and responsiveness to user guidance in model-generated text. SHS is explicitly not an automatic hallucination detector or benchmark metric; instead, it captures how hallucination phenomena manifest from a user perspective under realistic interaction conditions. A real-world evaluation with 210 participants demonstrates high clarity, coherent response behavior, and construct validity, supported by statistical analysis including internal consistency (Cronbach's alpha = 0.87$) and significant inter-dimension correlations (p < 0.001$). Comparative analysis with SUS and SCS reveals complementary measurement properties, supporting SHS as a practical tool for comparative analysis, iterative system development, and deployment monitoring.