Multi-Microphone Speech Emotion Recognition using the Hierarchical Token-semantic Audio Transformer Architecture

TL;DR

This paper introduces a multi-microphone speech emotion recognition system using a hierarchical token-semantic audio transformer, significantly improving robustness and accuracy in reverberant, real-world environments.

Contribution

It presents a novel multi-microphone processing approach combined with a transformer architecture to enhance emotion recognition in challenging acoustic conditions.

Findings

Multi-microphone input improves emotion classification accuracy.

Transformer-based model outperforms single-channel baselines.

Averaging and summing strategies effectively utilize multi-channel data.

Abstract

The performance of most emotion recognition systems degrades in real-life situations ('in the wild' scenarios) where the audio is contaminated by reverberation. Our study explores new methods to alleviate the performance degradation of SER algorithms and develop a more robust system for adverse conditions. We propose processing multi-microphone signals to address these challenges and improve emotion classification accuracy. We adopt a state-of-the-art transformer model, the HTS-AT, to handle multi-channel audio inputs. We evaluate two strategies: averaging mel-spectrograms across channels and summing patch-embedded representations. Our multi-microphone model achieves superior performance compared to single-channel baselines when tested on real-world reverberant environments.