Evaluating an LLM-Powered Chatbot for Cognitive Restructuring: Insights from Mental Health Professionals

TL;DR

This study evaluates an LLM-powered chatbot for cognitive restructuring in mental health therapy, highlighting its capabilities and limitations through user interactions and professional review, emphasizing the need for oversight.

Contribution

It provides empirical insights into the practical application of LLMs in psychotherapy, identifying both potential benefits and critical challenges for safe deployment.

Findings

LLM chatbot can follow core CR protocols and prompt Socratic questioning

Issues include power imbalance, advice-giving, and misunderstood cues

Highlights need for expert oversight to ensure safety and effectiveness

Abstract

Recent advancements in large language models (LLMs) promise to expand mental health interventions by emulating therapeutic techniques, potentially easing barriers to care. Yet there is a lack of real-world empirical evidence evaluating the strengths and limitations of LLM-enabled psychotherapy interventions. In this work, we evaluate an LLM-powered chatbot, designed via prompt engineering to deliver cognitive restructuring (CR), with 19 users. Mental health professionals then examined the resulting conversation logs to uncover potential benefits and pitfalls. Our findings indicate that an LLM-based CR approach has the capability to adhere to core CR protocols, prompt Socratic questioning, and provide empathetic validation. However, issues of power imbalances, advice-giving, misunderstood cues, and excessive positivity reveal deeper challenges, including the potential to erode therapeutic rapport and ethical concerns. We also discuss design implications for leveraging LLMs in psychotherapy and underscore the importance of expert oversight to mitigate these concerns, which are critical steps toward safer, more effective AI-assisted interventions.