Investigating Large Language Models' Perception of Emotion Using Appraisal Theory

TL;DR

This study examines how large language models perceive emotions using appraisal theory, revealing similarities to humans in response dynamics but differences in key appraisal dimensions and response magnitudes, and highlighting their sensitivity to question framing.

Contribution

It applies a validated clinical instrument to assess LLMs' emotion perception, providing new insights into their psychological understanding and response patterns.

Findings

LLMs show similar emotion response dynamics to humans.

Responses do not differ along key appraisal dimensions as predicted.

GPTs are sensitive to question phrasing and instructions.

Abstract

Large Language Models (LLM) like ChatGPT have significantly advanced in recent years and are now being used by the general public. As more people interact with these systems, improving our understanding of these black box models is crucial, especially regarding their understanding of human psychological aspects. In this work, we investigate their emotion perception through the lens of appraisal and coping theory using the Stress and Coping Process Questionaire (SCPQ). SCPQ is a validated clinical instrument consisting of multiple stories that evolve over time and differ in key appraisal variables such as controllability and changeability. We applied SCPQ to three recent LLMs from OpenAI, davinci-003, ChatGPT, and GPT-4 and compared the results with predictions from the appraisal theory and human data. The results show that LLMs' responses are similar to humans in terms of dynamics of appraisal and coping, but their responses did not differ along key appraisal dimensions as predicted by the theory and data. The magnitude of their responses is also quite different from humans in several variables. We also found that GPTs can be quite sensitive to instruction and how questions are asked. This work adds to the growing literature evaluating the psychological aspects of LLMs and helps enrich our understanding of the current models.