Exploring The Design of Prompts For Applying GPT-3 based Chatbots: A Mental Wellbeing Case Study on Mechanical Turk

TL;DR

This paper presents a case study on designing prompts for GPT-3 chatbots aimed at mental wellbeing, involving a large-scale experiment to evaluate prompt dimensions and user perceptions.

Contribution

It introduces a systematic approach to prompt design for GPT-3 chatbots in a mental health context, with experimental evaluation and analysis.

Findings

Prompt dimensions significantly affect user perceptions

Participants reported improved mood management

Design guidelines for task-specific GPT-3 chatbots

Abstract

Large-Language Models like GPT-3 have the potential to enable HCI designers and researchers to create more human-like and helpful chatbots for specific applications. But evaluating the feasibility of these chatbots and designing prompts that optimize GPT-3 for a specific task is challenging. We present a case study in tackling these questions, applying GPT-3 to a brief 5-minute chatbot that anyone can talk to better manage their mood. We report a randomized factorial experiment with 945 participants on Mechanical Turk that tests three dimensions of prompt design to initialize the chatbot (identity, intent, and behaviour), and present both quantitative and qualitative analyses of conversations and user perceptions of the chatbot. We hope other HCI designers and researchers can build on this case study, for other applications of GPT-3 based chatbots to specific tasks, and build on and extend the methods we use for prompt design, and evaluation of the prompt design.