EmoPrefer: Can Large Language Models Understand Human Emotion Preferences?

TL;DR

This paper introduces EmoPrefer, a novel approach leveraging multimodal large language models to understand human emotion preferences, including a new dataset and benchmark for evaluating model performance in emotion preference prediction.

Contribution

It is the first work to explore LLMs for decoding human emotion preferences, providing a new dataset, benchmark, and strategies to improve preference prediction accuracy.

Findings

MLLMs can effectively predict human emotion preferences.

The EmoPrefer-Bench evaluates various models and prompting techniques.

Proposed strategies improve model performance in emotion preference understanding.

Abstract

Descriptive Multimodal Emotion Recognition (DMER) has garnered increasing research attention. Unlike traditional discriminative paradigms that rely on predefined emotion taxonomies, DMER aims to describe human emotional state using free-form natural language, enabling finer-grained and more interpretable emotion representations. However, this free-form prediction paradigm introduces new challenges regarding its evaluation. Previous works depend on ground-truth descriptions, but emotions are inherently tied to diverse human behaviors, and generating a comprehensive and accurate description is inherently demanding. Other researchers reformulate this problem into a more tractable human preference learning task, but pairwise preference annotation involves substantial manual effort. This leads to a question: can we leverage multimodal LLMs (MLLMs) to achieve more cost-efficient preference annotation? To answer this, we propose EmoPrefer, a pioneering work exploring the potential of LLMs in decoding human emotion preferences. Specifically, we construct the first emotion preference dataset, EmoPrefer-Data, featuring high-quality preference annotations from experts. Additionally, we introduce EmoPrefer-Bench, which evaluates the performance of various MLLMs and prompting techniques in preference prediction, while also revealing new strategies to enhance their performance. To the best of our knowledge, this is the first work exploring the capabilities of LLMs in understanding human emotion preferences. Our work advances the field of DMER and lays the foundation for more intelligent human-computer interaction.