CARE-Bench: A Benchmark of Diverse Client Simulations Guided by Expert Principles for Evaluating LLMs in Psychological Counseling

TL;DR

CARE-Bench is a new benchmark for evaluating LLMs in psychological counseling, using diverse simulated clients and multidimensional metrics to better assess model performance and guide future improvements.

Contribution

It introduces CARE-Bench, a dynamic, expert-guided client simulation benchmark with multidimensional evaluation for assessing counseling capabilities of LLMs.

Findings

Current LLMs show limitations in handling diverse clients.

CARE-Bench reveals specific weaknesses in existing models.

Analysis guides development of more effective counseling models.

Abstract

The mismatch between the growing demand for psychological counseling and the limited availability of services has motivated research into the application of Large Language Models (LLMs) in this domain. Consequently, there is a need for a robust and unified benchmark to assess the counseling competence of various LLMs. Existing works, however, are limited by unprofessional client simulation, static question-and-answer evaluation formats, and unidimensional metrics. These limitations hinder their effectiveness in assessing a model's comprehensive ability to handle diverse and complex clients. To address this gap, we introduce \textbf{CARE-Bench}, a dynamic and interactive automated benchmark. It is built upon diverse client profiles derived from real-world counseling cases and simulated according to expert guidelines. CARE-Bench provides a multidimensional performance evaluation grounded in established psychological scales. Using CARE-Bench, we evaluate several general-purpose LLMs and specialized counseling models, revealing their current limitations. In collaboration with psychologists, we conduct a detailed analysis of the reasons for LLMs' failures when interacting with clients of different types, which provides directions for developing more comprehensive, universal, and effective counseling models.