Toward Large Language Models as a Therapeutic Tool: Comparing Prompting Techniques to Improve GPT-Delivered Problem-Solving Therapy

TL;DR

This study investigates how prompt engineering can enhance Large Language Models' ability to deliver psychotherapy, specifically Problem-Solving Therapy, highlighting improvements and limitations in model performance and therapeutic quality.

Contribution

It is among the first to evaluate various prompting techniques for improving LLMs' psychotherapy delivery, focusing on quality, consistency, and empathy.

Findings

Prompt engineering improves therapy protocol adherence.

Models show increased empathy with optimized prompts.

Limitations remain in model consistency and nuanced understanding.

Abstract

While Large Language Models (LLMs) are being quickly adapted to many domains, including healthcare, their strengths and pitfalls remain under-explored. In our study, we examine the effects of prompt engineering to guide Large Language Models (LLMs) in delivering parts of a Problem-Solving Therapy (PST) session via text, particularly during the symptom identification and assessment phase for personalized goal setting. We present evaluation results of the models' performances by automatic metrics and experienced medical professionals. We demonstrate that the models' capability to deliver protocolized therapy can be improved with the proper use of prompt engineering methods, albeit with limitations. To our knowledge, this study is among the first to assess the effects of various prompting techniques in enhancing a generalist model's ability to deliver psychotherapy, focusing on overall quality, consistency, and empathy. Exploring LLMs' potential in delivering psychotherapy holds promise with the current shortage of mental health professionals amid significant needs, enhancing the potential utility of AI-based and AI-enhanced care services.