Emotional Theory of Mind: Bridging Fast Visual Processing with Slow Linguistic Reasoning
Yasaman Etesam, \"Ozge Nilay Yal\c{c}{\i}n, Chuxuan Zhang, Angelica, Lim

TL;DR
This paper introduces methods to enhance emotional theory of mind in AI by combining fast visual cues with slow linguistic reasoning, using narrative captions and multimodal models.
Contribution
It presents novel approaches to incorporate emotional reasoning through narrative captions and multimodal models, bridging visual perception and linguistic inference.
Findings
Combining visual cues with language models improves emotion understanding.
Zero-shot classifiers and fine-tuned models effectively generate emotion-related narratives.
The approach advances affective computing by integrating fast and slow processing.
Abstract
The emotional theory of mind problem requires facial expressions, body pose, contextual information and implicit commonsense knowledge to reason about the person's emotion and its causes, making it currently one of the most difficult problems in affective computing. In this work, we propose multiple methods to incorporate the emotional reasoning capabilities by constructing "narrative captions" relevant to emotion perception, that includes contextual and physical signal descriptors that focuses on "Who", "What", "Where" and "How" questions related to the image and emotions of the individual. We propose two distinct ways to construct these captions using zero-shot classifiers (CLIP) and fine-tuning visual-language models (LLaVA) over human generated descriptors. We further utilize these captions to guide the reasoning of language (GPT-4) and vision-language models (LLaVa, GPT-Vision). We…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMultimodal Machine Learning Applications · Sentiment Analysis and Opinion Mining · Topic Modeling
