Can Large Language Models be Used to Provide Psychological Counselling? An Analysis of GPT-4-Generated Responses Using Role-play Dialogues
Michimasa Inaba, Mariko Ukiyo, Keiko Takamizo

TL;DR
This study evaluates GPT-4's ability to generate appropriate counseling responses in role-play scenarios, comparing its performance to human counselors to assess its potential for mental health support.
Contribution
It provides an empirical evaluation of GPT-4's responses in counseling dialogues, highlighting its competitiveness with human counselors in simulated scenarios.
Findings
GPT-4 responses were rated as competitive with human counselors.
Role-play data was annotated with counselor intentions for analysis.
The study suggests potential for large language models in mental health counseling.
Abstract
Mental health care poses an increasingly serious challenge to modern societies. In this context, there has been a surge in research that utilizes information technologies to address mental health problems, including those aiming to develop counseling dialogue systems. However, there is a need for more evaluations of the performance of counseling dialogue systems that use large language models. For this study, we collected counseling dialogue data via role-playing scenarios involving expert counselors, and the utterances were annotated with the intentions of the counselors. To determine the feasibility of a dialogue system in real-world counseling scenarios, third-party counselors evaluated the appropriateness of responses from human counselors and those generated by GPT-4 in identical contexts in role-play dialogue data. Analysis of the evaluation results showed that the responses…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Mental Health via Writing · Artificial Intelligence in Healthcare and Education
MethodsLinear Layer · Dense Connections · Label Smoothing · Adam · Attention Is All You Need · Softmax · Multi-Head Attention · Layer Normalization · Residual Connection · Absolute Position Encodings
