LLM-empowered Chatbots for Psychiatrist and Patient Simulation: Application and Evaluation

TL;DR

This paper explores the development and evaluation of ChatGPT-powered chatbots for simulating psychiatrist-patient interactions, demonstrating their feasibility and analyzing how prompt design affects performance and user experience.

Contribution

It introduces a novel application of ChatGPT in psychiatric outpatient simulation and provides an iterative development process aligned with real-world clinical scenarios.

Findings

ChatGPT chatbots are feasible for psychiatric simulation.

Prompt design significantly influences chatbot behavior.

User experience varies with different prompt strategies.

Abstract

Empowering chatbots in the field of mental health is receiving increasing amount of attention, while there still lacks exploration in developing and evaluating chatbots in psychiatric outpatient scenarios. In this work, we focus on exploring the potential of ChatGPT in powering chatbots for psychiatrist and patient simulation. We collaborate with psychiatrists to identify objectives and iteratively develop the dialogue system to closely align with real-world scenarios. In the evaluation experiments, we recruit real psychiatrists and patients to engage in diagnostic conversations with the chatbots, collecting their ratings for assessment. Our findings demonstrate the feasibility of using ChatGPT-powered chatbots in psychiatric scenarios and explore the impact of prompt designs on chatbot behavior and user experience.