MHDash: An Online Platform for Benchmarking Mental Health-Aware AI Assistants

TL;DR

MHDash is an open-source platform that enables detailed, risk-aware evaluation of mental health AI assistants, revealing nuanced performance issues especially in high-risk and multi-turn scenarios.

Contribution

We introduce MHDash, a comprehensive platform for developing, evaluating, and auditing mental health AI systems with fine-grained, risk-sensitive analysis capabilities.

Findings

High-risk cases show divergence among LLMs despite similar overall accuracy.

Some models rank severity consistently but fail in absolute risk detection.

Performance drops in multi-turn dialogues where risk signals are subtle.

Abstract

Large language models (LLMs) are increasingly applied in mental health support systems, where reliable recognition of high-risk states such as suicidal ideation and self-harm is safety-critical. However, existing evaluations primarily rely on aggregate performance metrics, which often obscure risk-specific failure modes and provide limited insight into model behavior in realistic, multi-turn interactions. We present MHDash, an open-source platform designed to support the development, evaluation, and auditing of AI systems for mental health applications. MHDash integrates data collection, structured annotation, multi-turn dialogue generation, and baseline evaluation into a unified pipeline. The platform supports annotations across multiple dimensions, including Concern Type, Risk Level, and Dialogue Intent, enabling fine-grained and risk-aware analysis. Our results reveal several key findings: (i) simple baselines and advanced LLM APIs exhibit comparable overall accuracy yet diverge significantly on high-risk cases; (ii) some LLMs maintain consistent ordinal severity ranking while failing absolute risk classification, whereas others achieve reasonable aggregate scores but suffer from high false negative rates on severe categories; and (iii) performance gaps are amplified in multi-turn dialogues, where risk signals emerge gradually. These observations demonstrate that conventional benchmarks are insufficient for safety-critical mental health settings. By releasing MHDash as an open platform, we aim to promote reproducible research, transparent evaluation, and safety-aligned development of AI systems for mental health support.