Multimodal Large Language Models for End-to-End Affective Computing: Benchmarking and Boosting with Generative Knowledge Prompting

TL;DR

This paper benchmarks multimodal large language models for affective computing, analyzes their performance variability, and introduces a generative knowledge prompting strategy that significantly boosts their ability to interpret human emotions.

Contribution

It provides a systematic evaluation of open-source MLLMs on MAC tasks and proposes a novel hybrid prompting and fine-tuning method to enhance their performance.

Findings

Benchmarking reveals performance gaps across models and datasets.

Hybrid prompting and fine-tuning improve MAC task accuracy.

Insights into architecture and data influence on affective analysis.

Abstract

Multimodal Affective Computing (MAC) aims to recognize and interpret human emotions by integrating information from diverse modalities such as text, video, and audio. Recent advancements in Multimodal Large Language Models (MLLMs) have significantly reshaped the landscape of MAC by offering a unified framework for processing and aligning cross-modal information. However, practical challenges remain, including performance variability across complex MAC tasks and insufficient understanding of how architectural designs and data characteristics impact affective analysis. To address these gaps, we conduct a systematic benchmark evaluation of state-of-the-art open-source MLLMs capable of concurrently processing audio, visual, and textual modalities across multiple established MAC datasets. Our evaluation not only compares the performance of these MLLMs but also provides actionable insights into model optimization by analyzing the influence of model architectures and dataset properties. Furthermore, we propose a novel hybrid strategy that combines generative knowledge prompting with supervised fine-tuning to enhance MLLMs' affective computing capabilities. Experimental results demonstrate that this integrated approach significantly improves performance across various MAC tasks, offering a promising avenue for future research and development in this field. Our code is released on https://github.com/LuoMSen/MLLM-MAC.