An Evaluation of Generative Pre-Training Model-based Therapy Chatbot for Caregivers

TL;DR

This study evaluates a GPT-2 based therapy chatbot for caregivers, fine-tuned on therapy transcripts, revealing strengths and limitations in generating human-like, emotionally appropriate responses for mental health support.

Contribution

It demonstrates the application of fine-tuned GPT-2 models in therapy chatbots and provides an analysis of their conversational qualities compared to human therapists.

Findings

Fine-tuned model produces more non-word outputs.

Generated responses are closer in length to therapist responses.

Both models tend to produce more negative than positive outputs.

Abstract

With the advent of off-the-shelf intelligent home products and broader internet adoption, researchers increasingly explore smart computing applications that provide easier access to health and wellness resources. AI-based systems like chatbots have the potential to provide services that could provide mental health support. However, existing therapy chatbots are often retrieval-based, requiring users to respond with a constrained set of answers, which may not be appropriate given that such pre-determined inquiries may not reflect each patient's unique circumstances. Generative-based approaches, such as the OpenAI GPT models, could allow for more dynamic conversations in therapy chatbot contexts than previous approaches. To investigate the generative-based model's potential in therapy chatbot contexts, we built a chatbot using the GPT-2 model. We fine-tuned it with 306 therapy session transcripts between family caregivers of individuals with dementia and therapists conducting Problem Solving Therapy. We then evaluated the model's pre-trained and the fine-tuned model in terms of basic qualities using three meta-information measurements: the proportion of non-word outputs, the length of response, and sentiment components. Results showed that: (1) the fine-tuned model created more non-word outputs than the pre-trained model; (2) the fine-tuned model generated outputs whose length was more similar to that of the therapists compared to the pre-trained model; (3) both the pre-trained model and fine-tuned model were likely to generate more negative and fewer positive outputs than the therapists. We discuss potential reasons for the problem, the implications, and solutions for developing therapy chatbots and call for investigations of the AI-based system application.