Team LEYA in 10th ABAW Competition: Multimodal Ambivalence/Hesitancy Recognition Approach

TL;DR

This paper presents a multimodal approach combining scene, face, audio, and text data for ambivalence and hesitancy recognition in videos, achieving significant improvements over unimodal methods in the ABAW Competition.

Contribution

The paper introduces a novel multimodal fusion framework that integrates four different modalities with prototype-augmented models for improved behavioral state recognition.

Findings

Multimodal fusion outperforms all unimodal baselines.

Best fusion model achieves 83.25% MF1 score.

Ensemble of models reaches 71.43% test accuracy.

Abstract

Ambivalence/hesitancy recognition in unconstrained videos is a challenging problem due to the subtle, multimodal, and context-dependent nature of this behavioral state. In this paper, a multimodal approach for video-level ambivalence/hesitancy recognition is presented for the 10th ABAW Competition. The proposed approach integrates four complementary modalities: scene, face, audio, and text. Scene dynamics are captured with a VideoMAE-based model, facial information is encoded through emotional frame-level embeddings aggregated by statistical pooling, acoustic representations are extracted with EmotionWav2Vec2.0 and processed by a Mamba-based temporal encoder, and linguistic cues are modeled using fine-tuned transformer-based text models. The resulting unimodal embeddings are further combined using multimodal fusion models, including prototype-augmented variants. Experiments on the BAH corpus demonstrate clear gains of multimodal fusion over all unimodal baselines. The best unimodal configuration achieved an average MF1 of 70.02%, whereas the best multimodal fusion model reached 83.25%. The highest final test performance, 71.43%, was obtained by an ensemble of five prototype-augmented fusion models. The obtained results highlight the importance of complementary multimodal cues and robust fusion strategies for ambivalence/hesitancy recognition.