Designing an LLM-Based Behavioral Activation Chatbot for Young People with Depression: Insights from an Evaluation with Artificial Users and Clinical Experts

TL;DR

This study evaluates an LLM-based behavioral activation chatbot for depression, demonstrating it can reliably deliver interventions but faces challenges in clinical judgment, highlighting areas for future improvement.

Contribution

Introduces a novel LLM-based depression chatbot evaluated with validated fidelity measures, providing insights into its strengths and limitations in clinical reasoning.

Findings

Chatbot reliably executes behavioral activation protocols.

Maintains safety protocols during interactions.

Struggles with clinical judgment and activity feasibility verification.

Abstract

LLMs promise to overcome limitations of rule-based mental health chatbots through improved natural language capabilities, yet their ability to deliver evidence-based psychological interventions remains largely unverified because evaluations rarely apply the validated fidelity measures used to assess psychotherapists. We developed an LLM-based chatbot that delivers behavioral activation for depression and generated 48 complete chat sessions with diverse artificial users. Ten psychotherapists assessed these sessions using the Quality of Behavioral Activation Scale (Q-BAS), a validated fidelity instrument. Results show that the chatbot reliably executed the intervention across all phases and maintained safety protocols, but it struggled with clinical judgment, particularly when verifying the feasibility of proposed activities. Overall, our findings suggest that LLM-based chatbots can execute therapeutic protocols with high fidelity, while robust clinical reasoning remains an open challenge. We outline design implications to address this gap and provide the chatbot and artificial user prompts.