On Prompt Sensitivity of ChatGPT in Affective Computing

TL;DR

This paper investigates how prompt variations and generation parameters influence ChatGPT's performance in affective computing tasks like sentiment, toxicity, and sarcasm detection, highlighting sensitivity to prompt design.

Contribution

It introduces a systematic evaluation method for prompt sensitivity in ChatGPT across multiple affective computing tasks, exploring parameter effects and prompting strategies.

Findings

Performance varies significantly with prompt design and parameters.

Certain prompting strategies improve task-specific accuracy.

Model's adherence to instructions affects downstream usability.

Abstract

Recent studies have demonstrated the emerging capabilities of foundation models like ChatGPT in several fields, including affective computing. However, accessing these emerging capabilities is facilitated through prompt engineering. Despite the existence of some prompting techniques, the field is still rapidly evolving and many prompting ideas still require investigation. In this work, we introduce a method to evaluate and investigate the sensitivity of the performance of foundation models based on different prompts or generation parameters. We perform our evaluation on ChatGPT within the scope of affective computing on three major problems, namely sentiment analysis, toxicity detection, and sarcasm detection. First, we carry out a sensitivity analysis on pivotal parameters in auto-regressive text generation, specifically the temperature parameter $T$ and the top-$p$ parameter in Nucleus sampling, dictating how conservative or creative the model should be during generation. Furthermore, we explore the efficacy of several prompting ideas, where we explore how giving different incentives or structures affect the performance. Our evaluation takes into consideration performance measures on the affective computing tasks, and the effectiveness of the model to follow the stated instructions, hence generating easy-to-parse responses to be smoothly used in downstream applications.