Emotional Theory of Mind: Bridging Fast Visual Processing with Slow Linguistic Reasoning

TL;DR

This paper introduces methods to enhance emotional theory of mind in AI by combining fast visual cues with slow linguistic reasoning, using narrative captions and multimodal models.

Contribution

It presents novel approaches to incorporate emotional reasoning through narrative captions and multimodal models, bridging visual perception and linguistic inference.

Findings

Combining visual cues with language models improves emotion understanding.

Zero-shot classifiers and fine-tuned models effectively generate emotion-related narratives.

The approach advances affective computing by integrating fast and slow processing.

Abstract

The emotional theory of mind problem requires facial expressions, body pose, contextual information and implicit commonsense knowledge to reason about the person's emotion and its causes, making it currently one of the most difficult problems in affective computing. In this work, we propose multiple methods to incorporate the emotional reasoning capabilities by constructing "narrative captions" relevant to emotion perception, that includes contextual and physical signal descriptors that focuses on "Who", "What", "Where" and "How" questions related to the image and emotions of the individual. We propose two distinct ways to construct these captions using zero-shot classifiers (CLIP) and fine-tuning visual-language models (LLaVA) over human generated descriptors. We further utilize these captions to guide the reasoning of language (GPT-4) and vision-language models (LLaVa, GPT-Vision). We evaluate the use of the resulting models in an image-to-language-to-emotion task. Our experiments showed that combining the "Fast" narrative descriptors and "Slow" reasoning of language models is a promising way to achieve emotional theory of mind.