Assessing the Effectiveness of LLMs in Delivering Cognitive Behavioral Therapy

TL;DR

This study evaluates the potential of large language models to emulate cognitive behavioral therapy by analyzing their ability to generate therapeutic dialogues, highlighting their strengths and limitations in delivering mental health support.

Contribution

The paper introduces a comprehensive evaluation framework for LLMs in CBT, comparing generation-only and retrieval-augmented approaches with both proprietary and open-source models.

Findings

LLMs can generate CBT-like dialogues.

Models struggle with conveying empathy.

Models have limitations in maintaining consistency.

Abstract

As mental health issues continue to rise globally, there is an increasing demand for accessible and scalable therapeutic solutions. Many individuals currently seek support from Large Language Models (LLMs), even though these models have not been validated for use in counseling services. In this paper, we evaluate LLMs' ability to emulate professional therapists practicing Cognitive Behavioral Therapy (CBT). Using anonymized, transcribed role-play sessions between licensed therapists and clients, we compare two approaches: (1) a generation-only method and (2) a Retrieval-Augmented Generation (RAG) approach using CBT guidelines. We evaluate both proprietary and open-source models for linguistic quality, semantic coherence, and therapeutic fidelity using standard natural language generation (NLG) metrics, natural language inference (NLI), and automated scoring for skills assessment. Our results indicate that while LLMs can generate CBT-like dialogues, they are limited in their ability to convey empathy and maintain consistency.