Multimodal Emotion Recognition using Audio-Video Transformer Fusion with Cross Attention

TL;DR

This paper introduces AVT-CA, a novel transformer-based multimodal emotion recognition model that effectively fuses audio and visual cues using cross attention and hierarchical feature refinement, leading to improved accuracy.

Contribution

The paper presents a hierarchical video feature extraction method combined with cross-attention fusion in a transformer architecture for robust multimodal emotion recognition.

Findings

AVT-CA outperforms state-of-the-art methods on benchmark datasets.

Significant improvements in accuracy and F1-score achieved.

Effective suppression of irrelevant information through hierarchical attention.

Abstract

Multimodal emotion recognition (MER) aims to infer human affect by jointly modeling audio and visual cues; however, existing approaches often struggle with temporal misalignment, weakly discriminative feature representations, and suboptimal fusion of heterogeneous modalities. To address these challenges, we propose AVT-CA, an Audio-Video Transformer architecture with cross attention for robust emotion recognition. The proposed model introduces a hierarchical video feature representation that combines channel attention, spatial attention, and local feature extraction to emphasize emotionally salient regions while suppressing irrelevant information. These refined visual features are integrated with audio representations through an intermediate transformer-based fusion mechanism that captures interlinked temporal dependencies across modalities. Furthermore, a cross-attention module selectively reinforces mutually consistent audio-visual cues, enabling effective feature selection and noise-aware fusion. Extensive experiments on three benchmark datasets, CMU-MOSEI, RAVDESS, and CREMA-D, demonstrate that AVT-CA consistently outperforms state-of-the-art baselines, achieving significant improvements in both accuracy and F1-score. Our source code is publicly available at https://github.com/shravan-18/AVTCA.