MMAFFBen: A Multilingual and Multimodal Affective Analysis Benchmark for Evaluating LLMs and VLMs

TL;DR

This paper introduces MMAFFBen, a comprehensive multilingual multimodal benchmark for affective analysis, evaluating the capabilities of large models across text, image, and video modalities in 35 languages.

Contribution

It presents the first extensive open-source benchmark and datasets for evaluating and fine-tuning models on multilingual multimodal affective analysis tasks.

Findings

Systematic evaluation of LMs on affective tasks across multiple modalities.

Development of datasets for fine-tuning affective analysis models.

Comparison of model performances, including GPT-4o-mini.

Abstract

Large language models and vision-language models (which we jointly call LMs) have transformed NLP and CV, demonstrating remarkable potential across various fields. However, their capabilities in affective analysis (i.e. sentiment analysis and emotion detection) remain underexplored. This gap is largely due to the absence of comprehensive evaluation benchmarks, and the inherent complexity of affective analysis tasks. In this paper, we introduce MMAFFBen, the first extensive open-source benchmark for multilingual multimodal affective analysis. MMAFFBen encompasses text, image, and video modalities across 35 languages, covering four key affective analysis tasks: sentiment polarity, sentiment intensity, emotion classification, and emotion intensity. Moreover, we construct the MMAFFIn dataset for fine-tuning LMs on affective analysis tasks, and further develop MMAFFLM-3b and MMAFFLM-7b based on it. We evaluate various representative LMs, including GPT-4o-mini, providing a systematic comparison of their affective understanding capabilities. This project is available at https://github.com/lzw108/MMAFFBen.