OmniMER: Auxiliary-Enhanced LLM Adaptation for Indonesian Multimodal Emotion Recognition

TL;DR

This paper introduces IndoMER, a new multimodal Indonesian emotion recognition dataset, and proposes OmniMER, an auxiliary-enhanced adaptation framework that improves emotion recognition accuracy by leveraging modality-specific auxiliary tasks.

Contribution

The paper presents the first Indonesian multimodal emotion recognition benchmark and a novel adaptation framework that enhances model performance through auxiliary modality-specific perception tasks.

Findings

OmniMER outperforms the base model with significant F1 score improvements.

The dataset captures realistic challenges like cross-modal inconsistency and long-tailed distributions.

Cross-lingual evaluation shows the framework's generalizability.

Abstract

Indonesian, spoken by over 200 million people, remains underserved in multimodal emotion recognition research despite its dominant presence on Southeast Asian social media platforms. We introduce IndoMER, the first multimodal emotion recognition benchmark for Indonesian, comprising 1,944 video segments from 203 speakers with temporally aligned text, audio, and visual annotations across seven emotion categories. The dataset exhibits realistic challenges including cross-modal inconsistency and long-tailed class distributions shaped by Indonesian cultural communication norms. To address these challenges, we propose OmniMER, a multimodal adaptation framework built upon Qwen2.5-Omni that enhances emotion recognition through three auxiliary modality-specific perception tasks: emotion keyword extraction for text, facial expression analysis for video, and prosody analysis for audio. These auxiliary tasks help the model identify emotion-relevant cues in each modality before fusion, reducing reliance on spurious correlations in low-resource settings. Experiments on IndoMER show that OmniMER achieves 0.582 Macro-F1 on sentiment classification and 0.454 on emotion recognition, outperforming the base model by 7.6 and 22.1 absolute points respectively. Cross-lingual evaluation on the Chinese CH-SIMS dataset further demonstrates the generalizability of the proposed framework. The dataset and code are publicly available. https://github.com/yanxm01/INDOMER