Do Large Language Models Align with Core Mental Health Counseling Competencies?

TL;DR

This paper evaluates whether large language models can meet core mental health counseling competencies using a new benchmark, revealing they perform well in some areas but lack in empathy and ethical reasoning, highlighting the need for specialized models.

Contribution

Introduces CounselingBench, a novel benchmark for assessing LLMs on mental health counseling competencies, and provides a comprehensive evaluation of various models' strengths and weaknesses.

Findings

Frontier models surpass minimum thresholds but not expert-level performance.

Models perform well in Intake, Assessment & Diagnosis but struggle with empathy and ethics.

Medical LLMs do not outperform generalist models in accuracy, but offer better justifications.

Abstract

The rapid evolution of Large Language Models (LLMs) presents a promising solution to the global shortage of mental health professionals. However, their alignment with essential counseling competencies remains underexplored. We introduce CounselingBench, a novel NCMHCE-based benchmark evaluating 22 general-purpose and medical-finetuned LLMs across five key competencies. While frontier models surpass minimum aptitude thresholds, they fall short of expert-level performance, excelling in Intake, Assessment & Diagnosis but struggling with Core Counseling Attributes and Professional Practice & Ethics. Surprisingly, medical LLMs do not outperform generalist models in accuracy, though they provide slightly better justifications while making more context-related errors. These findings highlight the challenges of developing AI for mental health counseling, particularly in competencies requiring empathy and nuanced reasoning. Our results underscore the need for specialized, fine-tuned models aligned with core mental health counseling competencies and supported by human oversight before real-world deployment. Code and data associated with this manuscript can be found at: https://github.com/cuongnguyenx/CounselingBench