Reasoning Is Not All You Need: Examining LLMs for Multi-Turn Mental Health Conversations

TL;DR

This paper introduces MedAgent and MultiSenseEval frameworks to generate and evaluate multi-turn mental health conversations with LLMs, revealing current models' limitations in patient-centric communication and diagnostic capabilities.

Contribution

The paper presents a novel synthetic data generation framework, a new dataset, and a holistic evaluation method for assessing LLMs in multi-turn mental health dialogues.

Findings

Frontier reasoning models perform below expectations in patient-centric communication.

Models struggle with advanced diagnostic capabilities, scoring around 31%.

Performance varies based on patient's persona and decreases with more conversation turns.

Abstract

Limited access to mental healthcare, extended wait times, and increasing capabilities of Large Language Models (LLMs) has led individuals to turn to LLMs for fulfilling their mental health needs. However, examining the multi-turn mental health conversation capabilities of LLMs remains under-explored. Existing evaluation frameworks typically focus on diagnostic accuracy and win-rates and often overlook alignment with patient-specific goals, values, and personalities required for meaningful conversations. To address this, we introduce MedAgent, a novel framework for synthetically generating realistic, multi-turn mental health sensemaking conversations and use it to create the Mental Health Sensemaking Dialogue (MHSD) dataset, comprising over 2,200 patient-LLM conversations. Additionally, we present MultiSenseEval, a holistic framework to evaluate the multi-turn conversation abilities of LLMs in healthcare settings using human-centric criteria. Our findings reveal that frontier reasoning models yield below-par performance for patient-centric communication and struggle at advanced diagnostic capabilities with average score of 31%. Additionally, we observed variation in model performance based on patient's persona and performance drop with increasing turns in the conversation. Our work provides a comprehensive synthetic data generation framework, a dataset and evaluation framework for assessing LLMs in multi-turn mental health conversations.