Multi-Microphone and Multi-Modal Emotion Recognition in Reverberant Environment

TL;DR

This paper introduces a multi-modal emotion recognition system combining audio and video analysis, utilizing advanced neural networks to improve accuracy in reverberant environments, outperforming single-modal and single-microphone methods.

Contribution

The paper presents a novel multi-modal emotion recognition system that integrates multi-channel audio with video analysis using advanced neural networks, specifically designed for reverberant environments.

Findings

Multimodal approach outperforms uni-modal methods in accuracy.

Multi-microphone system surpasses single-microphone performance.

System effective in challenging acoustic conditions with reverberation.

Abstract

This paper presents a Multi-modal Emotion Recognition (MER) system designed to enhance emotion recognition accuracy in challenging acoustic conditions. Our approach combines a modified and extended Hierarchical Token-semantic Audio Transformer (HTS-AT) for multi-channel audio processing with an R(2+1)D Convolutional Neural Networks (CNN) model for video analysis. We evaluate our proposed method on a reverberated version of the Ryerson audio-visual database of emotional speech and song (RAVDESS) dataset using synthetic and real-world Room Impulse Responsess (RIRs). Our results demonstrate that integrating audio and video modalities yields superior performance compared to uni-modal approaches, especially in challenging acoustic conditions. Moreover, we show that the multimodal (audiovisual) approach that utilizes multiple microphones outperforms its single-microphone counterpart.