MindEval: Benchmarking Language Models on Multi-turn Mental Health Support

TL;DR

MindEval introduces a comprehensive benchmark for evaluating language models in multi-turn mental health therapy scenarios, addressing limitations of existing assessments and revealing current models' weaknesses in realistic therapeutic interactions.

Contribution

The paper presents MindEval, a novel, fully automated framework for assessing language models in complex, multi-turn mental health conversations, developed with clinical psychologist collaboration.

Findings

All evaluated models scored below 4 out of 6, indicating room for improvement.

Model reasoning and scale do not necessarily improve performance.

Models perform worse with longer interactions and severe symptoms.

Abstract

Demand for mental health support through AI chatbots is surging, though current systems present several limitations, like sycophancy or overvalidation, and reinforcement of maladaptive beliefs. A core obstacle to the creation of better systems is the scarcity of benchmarks that capture the complexity of real therapeutic interactions. Most existing benchmarks either only test clinical knowledge through multiple-choice questions or assess single responses in isolation. To bridge this gap, we present MindEval, a framework designed in collaboration with Ph.D-level Licensed Clinical Psychologists for automatically evaluating language models in realistic, multi-turn mental health therapy conversations. Through patient simulation and automatic evaluation with LLMs, our framework balances resistance to gaming with reproducibility via its fully automated, model-agnostic design. We begin by quantitatively validating the realism of our simulated patients against human-generated text and by demonstrating strong correlations between automatic and human expert judgments. Then, we evaluate 12 state-of-the-art LLMs and show that all models struggle, scoring below 4 out of 6, on average, with particular weaknesses in problematic AI-specific patterns of communication. Notably, reasoning capabilities and model scale do not guarantee better performance, and systems deteriorate with longer interactions or when supporting patients with severe symptoms. We release all code, prompts, and human evaluation data.