Can Large Language Models be Used to Provide Psychological Counselling? An Analysis of GPT-4-Generated Responses Using Role-play Dialogues

TL;DR

This study evaluates GPT-4's ability to generate appropriate counseling responses in role-play scenarios, comparing its performance to human counselors to assess its potential for mental health support.

Contribution

It provides an empirical evaluation of GPT-4's responses in counseling dialogues, highlighting its competitiveness with human counselors in simulated scenarios.

Findings

GPT-4 responses were rated as competitive with human counselors.

Role-play data was annotated with counselor intentions for analysis.

The study suggests potential for large language models in mental health counseling.

Abstract

Mental health care poses an increasingly serious challenge to modern societies. In this context, there has been a surge in research that utilizes information technologies to address mental health problems, including those aiming to develop counseling dialogue systems. However, there is a need for more evaluations of the performance of counseling dialogue systems that use large language models. For this study, we collected counseling dialogue data via role-playing scenarios involving expert counselors, and the utterances were annotated with the intentions of the counselors. To determine the feasibility of a dialogue system in real-world counseling scenarios, third-party counselors evaluated the appropriateness of responses from human counselors and those generated by GPT-4 in identical contexts in role-play dialogue data. Analysis of the evaluation results showed that the responses generated by GPT-4 were competitive with those of human counselors.