Large Vision-Language Models as Emotion Recognizers in Context Awareness

TL;DR

This paper investigates the use of Large Vision-Language Models for context-aware emotion recognition, exploring fine-tuning, zero-shot, few-shot, and reasoning-enhanced methods, demonstrating competitive performance with minimal training.

Contribution

It systematically evaluates LVLMs for CAER across multiple paradigms, including training-free in-context learning and reasoning-based approaches, highlighting their potential without extensive data.

Findings

LVLMs achieve competitive CAER performance across paradigms.

Few-shot settings show strong results, indicating minimal training sufficiency.

Incorporating Chain-of-Thought improves reasoning and interpretability.

Abstract

Context-aware emotion recognition (CAER) is a complex and significant task that requires perceiving emotions from various contextual cues. Previous approaches primarily focus on designing sophisticated architectures to extract emotional cues from images. However, their knowledge is confined to specific training datasets and may reflect the subjective emotional biases of the annotators. Furthermore, acquiring large amounts of labeled data is often challenging in real-world applications. In this paper, we systematically explore the potential of leveraging Large Vision-Language Models (LVLMs) to empower the CAER task from three paradigms: 1) We fine-tune LVLMs on two CAER datasets, which is the most common way to transfer large models to downstream tasks. 2) We design zero-shot and few-shot patterns to evaluate the performance of LVLMs in scenarios with limited data or even completely unseen. In this case, a training-free framework is proposed to fully exploit the In-Context Learning (ICL) capabilities of LVLMs. Specifically, we develop an image similarity-based ranking algorithm to retrieve examples; subsequently, the instructions, retrieved examples, and the test example are combined to feed LVLMs to obtain the corresponding sentiment judgment. 3) To leverage the rich knowledge base of LVLMs, we incorporate Chain-of-Thought (CoT) into our framework to enhance the model's reasoning ability and provide interpretable results. Extensive experiments and analyses demonstrate that LVLMs achieve competitive performance in the CAER task across different paradigms. Notably, the superior performance in few-shot settings indicates the feasibility of LVLMs for accomplishing specific tasks without extensive training.