Towards a Client-Centered Assessment of LLM Therapists by Client Simulation

TL;DR

This paper introduces ClientCAST, a novel client-centered assessment method using simulated clients to evaluate LLM therapists, addressing ethical and technical challenges in large-scale evaluation.

Contribution

It proposes a scalable, ethical approach for assessing LLM therapists through client simulation and develops metrics for session outcome, alliance, and feelings.

Findings

ClientCAST provides reliable assessments of LLM therapists.

Evaluation of multiple LLMs demonstrates varying performance levels.

The approach reduces ethical risks associated with human client simulation.

Abstract

Although there is a growing belief that LLMs can be used as therapists, exploring LLMs' capabilities and inefficacy, particularly from the client's perspective, is limited. This work focuses on a client-centered assessment of LLM therapists with the involvement of simulated clients, a standard approach in clinical medical education. However, there are two challenges when applying the approach to assess LLM therapists at scale. Ethically, asking humans to frequently mimic clients and exposing them to potentially harmful LLM outputs can be risky and unsafe. Technically, it can be difficult to consistently compare the performances of different LLM therapists interacting with the same client. To this end, we adopt LLMs to simulate clients and propose ClientCAST, a client-centered approach to assessing LLM therapists by client simulation. Specifically, the simulated client is utilized to interact with LLM therapists and complete questionnaires related to the interaction. Based on the questionnaire results, we assess LLM therapists from three client-centered aspects: session outcome, therapeutic alliance, and self-reported feelings. We conduct experiments to examine the reliability of ClientCAST and use it to evaluate LLMs therapists implemented by Claude-3, GPT-3.5, LLaMA3-70B, and Mixtral 8*7B. Codes are released at https://github.com/wangjs9/ClientCAST.