Nuanced Emotion Recognition Based on a Segment-based MLLM Framework Leveraging Qwen3-Omni for AH Detection

TL;DR

This paper introduces a segment-based multimodal large language model framework leveraging Qwen3-Omni for nuanced emotion recognition in videos, effectively capturing subtle psychological states like Ambivalence and Hesitancy with high accuracy.

Contribution

It proposes a novel segment-based approach combined with fine-tuned Multimodal Large Language Models to improve recognition of complex emotional states in videos.

Findings

Achieved 85.1% accuracy on the test set.

Outperformed existing benchmarks in emotion recognition.

Validated the effectiveness of multimodal LLMs in detecting nuanced psychological states.

Abstract

Emotion recognition in videos is a pivotal task in affective computing, where identifying subtle psychological states such as Ambivalence and Hesitancy holds significant value for behavioral intervention and digital health. Ambivalence and Hesitancy states often manifest through cross-modal inconsistencies such as discrepancies between facial expressions, vocal tones, and textual semantics, posing a substantial challenge for automated recognition. This paper proposes a recognition framework that integrates temporal segment modeling with Multimodal Large Language Models. To address computational efficiency and token constraints in long video processing, we employ a segment-based strategy, partitioning videos into short clips with a maximum duration of 5 seconds. We leverage the Qwen3-Omni-30B-A3B model, fine-tuned on the BAH dataset using LoRA and full-parameter strategies via the MS-Swift framework, enabling the model to synergistically analyze visual and auditory signals. Experimental results demonstrate that the proposed method achieves an accuracy of 85.1% on the test set, significantly outperforming existing benchmarks and validating the superior capability of Multimodal Large Language Models in capturing complex and nuanced emotional conflicts. The code is released at https://github.com/dlnn123/A-H-Detection-with-Qwen-Omni.git.