Beyond Simulations: What 20,000 Real Conversations Reveal About Mental Health AI Safety

TL;DR

This study compares safety test set performance with real-world data for mental health AI, revealing that real-world safety failures are fewer than test set failures, emphasizing the need for ongoing safety evaluation in deployment.

Contribution

It provides the first ecological audit of over 20,000 real conversations, highlighting discrepancies between test set results and real-world safety performance of mental health AI.

Findings

Purpose-built AI significantly reduces harmful content compared to general-purpose LLMs.

Test set failure rates are higher than real-world failure rates.

Real-world safety failures are rare, supporting continuous safety monitoring.

Abstract

Large language models (LLMs) are increasingly used for mental health support, yet existing safety evaluations rely primarily on small, simulation-based test sets that have an unknown relationship to the linguistic distribution of real usage. In this study, we present replications of four published safety test sets targeting suicide risk assessment, harmful content generation, refusal robustness, and adversarial jailbreaks for a leading frontier generic AI model alongside an AI purpose built for mental health support. We then propose and conduct an ecological audit on over 20,000 real-world user conversations with the purpose-built AI designed with layered suicide and non-suicidal self-injury (NSSI) safeguards to compare test set performance to real world performance. While the purpose-built AI was significantly less likely than general-purpose LLMs to produce enabling or harmful content across suicide/NSSI (.4-11.27% vs 29.0-54.4%), eating disorder (8.4% vs 54.0%), and substance use (9.9% vs 45.0%) benchmark prompts, test set failure rates for suicide/NSSI were far higher than in real-world deployment. Clinician review of flagged conversations from the ecological audit identified zero cases of suicide risk that failed to receive crisis resources. Across all 20,000 conversations, three mentions of NSSI risk (.015%) did not trigger a crisis intervention; among sessions flagged by the LLM judge, this corresponds to an end-to-end system false negative rate of .38%, providing a lower bound on real-world safety failures. These findings support a shift toward continuous, deployment-relevant safety assurance for AI mental-health systems rather than limited set benchmark certification.