AI Safety Training Can be Clinically Harmful

TL;DR

This study reveals that current large language models used in mental health support can cause psychological harm due to safety alignment failures, highlighting the need for rigorous multi-axis evaluation before deployment.

Contribution

It systematically evaluates LLMs in therapeutic scenarios, exposing safety and protocol fidelity failures, and proposes a comprehensive evaluation framework for AI mental health systems.

Findings

Models show high surface acknowledgment but poor therapeutic appropriateness at high severity.

RLHF safety alignment can disrupt therapeutic mechanisms and safety in mental health applications.

Proposes a five-axis evaluation framework for safe deployment of AI in mental health.

Abstract

Large language models are being deployed as mental health support agents at scale, yet only 16% of LLM-based chatbot interventions have undergone rigorous clinical efficacy testing, and simulations reveal psychological deterioration in over one-third of cases. We evaluate four generative models on 250 Prolonged Exposure (PE) therapy scenarios and 146 CBT cognitive restructuring exercises (plus 29 severity-escalated variants), scored by a three-judge LLM panel. All models scored near-perfectly on surface acknowledgment (~0.91-1.00) while therapeutic appropriateness collapsed to 0.22-0.33 at the highest severity for three of four models, with protocol fidelity reaching zero for two. Under CBT severity escalation, one model's task completeness dropped from 92% to 71% while the frontier model's safety-interference score fell from 0.99 to 0.61. We identify a systematic, modality-spanning failure: RLHF safety alignment disrupts the therapeutic mechanism of action by grounding patients during imaginal exposure, offering false reassurance, inserting crisis resources into controlled exercises, and refusing to challenge distorted cognitions mentioning self-harm in PE; and through task abandonment or safety-preamble insertion during CBT cognitive restructuring. These findings motivate a five-axis evaluation framework (protocol fidelity, hallucination risk, behavioral consistency, crisis safety, demographic robustness), mapped onto FDA SaMD and EU AI Act requirements. We argue that no AI mental health system should proceed to deployment without passing multi-axis evaluation across all five dimensions.