Beyond Context to Cognitive Appraisal: Emotion Reasoning as a Theory of Mind Benchmark for Large Language Models

TL;DR

This paper introduces a new evaluation dataset based on Cognitive Appraisal Theory to assess large language models' ability to reason about emotions and contexts, highlighting current limitations and the importance of psychological theories.

Contribution

It presents a specialized ToM dataset for emotion reasoning grounded in Cognitive Appraisal Theory, enabling assessment of LLMs' higher-order emotion inference capabilities.

Findings

LLMs can perform some emotion reasoning but are limited in associating outcomes with specific emotions.

Current models struggle with linking situational outcomes and appraisals to emotions.

The study emphasizes integrating psychological theories into LLM training and evaluation.

Abstract

Datasets used for emotion recognition tasks typically contain overt cues that can be used in predicting the emotions expressed in a text. However, one challenge is that texts sometimes contain covert contextual cues that are rich in affective semantics, which warrant higher-order reasoning abilities to infer emotional states, not simply the emotions conveyed. This study advances beyond surface-level perceptual features to investigate how large language models (LLMs) reason about others' emotional states using contextual information, within a Theory-of-Mind (ToM) framework. Grounded in Cognitive Appraisal Theory, we curate a specialized ToM evaluation dataset1 to assess both forward reasoning - from context to emotion- and backward reasoning - from emotion to inferred context. We showed that LLMs can reason to a certain extent, although they are poor at associating situational outcomes and appraisals with specific emotions. Our work highlights the need for psychological theories in the training and evaluation of LLMs in the context of emotion reasoning.