Detail-Enhanced Intra- and Inter-modal Interaction for Audio-Visual Emotion Recognition

TL;DR

This paper introduces DE-III, a novel network for audio-visual emotion recognition that leverages optical flow and attentive feature enhancement to better capture facial details and improve recognition accuracy.

Contribution

The paper proposes a new DE-III network incorporating optical flow and intra- and inter-modal attention modules for enhanced emotion recognition.

Findings

Outperforms existing methods on three benchmark datasets

Effectively captures facial state changes with optical flow

Improves feature discriminability for emotion recognition

Abstract

Capturing complex temporal relationships between video and audio modalities is vital for Audio-Visual Emotion Recognition (AVER). However, existing methods lack attention to local details, such as facial state changes between video frames, which can reduce the discriminability of features and thus lower recognition accuracy. In this paper, we propose a Detail-Enhanced Intra- and Inter-modal Interaction network(DE-III) for AVER, incorporating several novel aspects. We introduce optical flow information to enrich video representations with texture details that better capture facial state changes. A fusion module integrates the optical flow estimation with the corresponding video frames to enhance the representation of facial texture variations. We also design attentive intra- and inter-modal feature enhancement modules to further improve the richness and discriminability of video and audio representations. A detailed quantitative evaluation shows that our proposed model outperforms all existing methods on three benchmark datasets for both concrete and continuous emotion recognition. To encourage further research and ensure replicability, we will release our full code upon acceptance.