On the Effectiveness of Creating Conversational Agent Personalities Through Prompting

TL;DR

This study evaluates how effectively prompt engineering can create distinct personalities in GPT-based conversational agents, analyzing linguistic features to identify differences across archetypes and suggesting the need for improved prompting strategies.

Contribution

It demonstrates the use of linguistic analysis to measure personality differences in GPT agents and highlights the limitations of current prompting methods for persistent personality creation.

Findings

Significant linguistic differences in GPT-3.5 and GPT-4 agents across multiple cues.

More cues distinguished in GPT-4 than GPT-3.5, indicating improved differentiation.

Current prompting approaches may require refinement for better personality consistency.

Abstract

In this work, we report on the effectiveness of our efforts to tailor the personality and conversational style of a conversational agent based on GPT-3.5 and GPT-4 through prompts. We use three personality dimensions with two levels each to create eight conversational agents archetypes. Ten conversations were collected per chatbot, of ten exchanges each, generating 1600 exchanges across GPT-3.5 and GPT-4. Using Linguistic Inquiry and Word Count (LIWC) analysis, we compared the eight agents on language elements including clout, authenticity, and emotion. Four language cues were significantly distinguishing in GPT-3.5, while twelve were distinguishing in GPT-4. With thirteen out of a total nineteen cues in LIWC appearing as significantly distinguishing, our results suggest possible novel prompting approaches may be needed to better suit the creation and evaluation of persistent conversational agent personalities or language styles.