Contextual Emotion Estimation from Image Captions

TL;DR

This paper investigates whether large language models can estimate human emotions from image captions, demonstrating that GPT-3.5 can reasonably predict emotions based on scene descriptions, offering a new approach to emotion estimation.

Contribution

The study introduces a novel method of using image captions and LLMs for emotion estimation, including a set of natural language descriptors and a dataset of annotated images.

Findings

GPT-3.5 provides reasonable emotion predictions

Caption-based approach offers interpretability for emotion perception

Accuracy varies depending on the emotion concept

Abstract

Emotion estimation in images is a challenging task, typically using computer vision methods to directly estimate people's emotions using face, body pose and contextual cues. In this paper, we explore whether Large Language Models (LLMs) can support the contextual emotion estimation task, by first captioning images, then using an LLM for inference. First, we must understand: how well do LLMs perceive human emotions? And which parts of the information enable them to determine emotions? One initial challenge is to construct a caption that describes a person within a scene with information relevant for emotion perception. Towards this goal, we propose a set of natural language descriptors for faces, bodies, interactions, and environments. We use them to manually generate captions and emotion annotations for a subset of 331 images from the EMOTIC dataset. These captions offer an interpretable representation for emotion estimation, towards understanding how elements of a scene affect emotion perception in LLMs and beyond. Secondly, we test the capability of a large language model to infer an emotion from the resulting image captions. We find that GPT-3.5, specifically the text-davinci-003 model, provides surprisingly reasonable emotion predictions consistent with human annotations, but accuracy can depend on the emotion concept. Overall, the results suggest promise in the image captioning and LLM approach.