Disentangling Prompt Element Level Risk Factors for Hallucinations and Omissions in Mental Health LLM Responses

TL;DR

This study introduces the UTCO framework to systematically evaluate mental health LLM responses, revealing that context and tone significantly influence hallucinations and omissions, especially in crisis scenarios.

Contribution

The paper presents UTCO, a novel prompt construction framework for stress testing mental health LLMs, highlighting the importance of dynamic evaluation over static benchmarks.

Findings

Hallucinations occurred in 6.5% of responses.

Omissions occurred in 13.2%, mainly in crisis prompts.

Context and tone are key factors influencing failures.

Abstract

Mental health concerns are often expressed outside clinical settings, including in high-distress help seeking, where safety-critical guidance may be needed. Consumer health informatics systems increasingly incorporate large language models (LLMs) for mental health question answering, yet many evaluations underrepresent narrative, high-distress inquiries. We introduce UTCO (User, Topic, Context, Tone), a prompt construction framework that represents an inquiry as four controllable elements for systematic stress testing. Using 2,075 UTCO-generated prompts, we evaluated Llama 3.3 and annotated hallucinations (fabricated or incorrect clinical content) and omissions (missing clinically necessary or safety-critical guidance). Hallucinations occurred in 6.5% of responses and omissions in 13.2%, with omissions concentrated in crisis and suicidal ideation prompts. Across regression, element-specific matching, and similarity-matched comparisons, failures were most consistently associated with context and tone, while user-background indicators showed no systematic differences after balancing. These findings support evaluating omissions as a primary safety outcome and moving beyond static benchmark question sets.