Multi-modal Conditional Attention Fusion for Dimensional Emotion Prediction

TL;DR

This paper introduces a novel multi-modal fusion approach called conditional attention fusion for continuous dimensional emotion prediction, dynamically weighting modalities based on current input and history, leading to improved performance.

Contribution

The paper proposes a dynamic, input-dependent fusion strategy using conditional attention, enhancing multimodal emotion prediction over traditional fixed-weight methods.

Findings

Outperforms common fusion strategies on AVEC2015 dataset

Effectively captures long-term dependencies with LSTM-RNN

Improves valence prediction accuracy

Abstract

Continuous dimensional emotion prediction is a challenging task where the fusion of various modalities usually achieves state-of-the-art performance such as early fusion or late fusion. In this paper, we propose a novel multi-modal fusion strategy named conditional attention fusion, which can dynamically pay attention to different modalities at each time step. Long-short term memory recurrent neural networks (LSTM-RNN) is applied as the basic uni-modality model to capture long time dependencies. The weights assigned to different modalities are automatically decided by the current input features and recent history information rather than being fixed at any kinds of situation. Our experimental results on a benchmark dataset AVEC2015 show the effectiveness of our method which outperforms several common fusion strategies for valence prediction.