Zero-shot Emotion Annotation in Facial Images Using Large Multimodal Models: Benchmarking and Prospects for Multi-Class, Multi-Frame Approaches

TL;DR

This paper evaluates the use of large multimodal models for zero-shot facial emotion annotation, demonstrating moderate accuracy and exploring multi-frame strategies to improve performance and reduce costs.

Contribution

It introduces a benchmark for zero-shot emotion annotation using LMMs and investigates multi-frame approaches for enhanced accuracy in complex scenarios.

Findings

Achieved ~50% precision in seven-class emotion classification

Improved to ~64% precision in ternary classification

Multi-frame integration slightly enhances annotation accuracy

Abstract

This study investigates the feasibility and performance of using large multimodal models (LMMs) to automatically annotate human emotions in everyday scenarios. We conducted experiments on the DailyLife subset of the publicly available FERV39k dataset, employing the GPT-4o-mini model for rapid, zero-shot labeling of key frames extracted from video segments. Under a seven-class emotion taxonomy ("Angry," "Disgust," "Fear," "Happy," "Neutral," "Sad," "Surprise"), the LMM achieved an average precision of approximately 50%. In contrast, when limited to ternary emotion classification (negative/neutral/positive), the average precision increased to approximately 64%. Additionally, we explored a strategy that integrates multiple frames within 1-2 second video clips to enhance labeling performance and reduce costs. The results indicate that this approach can slightly improve annotation accuracy. Overall, our preliminary findings highlight the potential application of zero-shot LMMs in human facial emotion annotation tasks, offering new avenues for reducing labeling costs and broadening the applicability of LMMs in complex multimodal environments.