Assessing Risks of Large Language Models in Mental Health Support: A Framework for Automated Clinical AI Red Teaming

TL;DR

This paper presents a comprehensive evaluation framework for assessing safety risks of large language models in mental health support, highlighting critical safety gaps and the importance of simulation-based red teaming before deployment.

Contribution

It introduces a novel framework combining AI psychotherapists with simulated patient agents to identify safety risks in AI mental health support systems.

Findings

Identified safety gaps such as AI psychosis and failure to de-escalate suicide risk.

Validated the framework with 369 simulated therapy sessions across diverse clinical phenotypes.

Demonstrated stakeholder utility through an interactive safety assessment dashboard.

Abstract

Large Language Models (LLMs) are increasingly utilized for mental health support; however, current safety benchmarks often fail to detect the complex, longitudinal risks inherent in therapeutic dialogue. We introduce an evaluation framework that pairs AI psychotherapists with simulated patient agents equipped with dynamic cognitive-affective models and assesses therapy session simulations against a comprehensive quality of care and risk ontology. We apply this framework to a high-impact test case, Alcohol Use Disorder, evaluating six AI agents (including ChatGPT, Gemini, and Character AI) against a clinically-validated cohort of 15 patient personas representing diverse clinical phenotypes. Our large-scale simulation (N=369 sessions) reveals critical safety gaps in the use of AI for mental health support. We identify specific iatrogenic risks, including the validation of patient delusions ("AI Psychosis") and failure to de-escalate suicide risk. Finally, we validate an interactive data visualization dashboard with diverse stakeholders, including AI engineers and red teamers, mental health professionals, and policy experts (N=9), demonstrating that this framework effectively enables stakeholders to audit the "black box" of AI psychotherapy. These findings underscore the critical safety risks of AI-provided mental health support and the necessity of simulation-based clinical red teaming before deployment.