How Attention Shapes Emotion: A Comparative Study of Attention Mechanisms for Speech Emotion Recognition

TL;DR

This paper benchmarks various optimized attention mechanisms for Speech Emotion Recognition, highlighting the trade-off between accuracy and efficiency, and providing insights for scalable SER system design.

Contribution

It systematically compares multiple efficient attention variants for SER, demonstrating their scalability benefits over standard self-attention.

Findings

Standard self-attention achieves highest accuracy.

Efficient attention variants significantly reduce latency and memory usage.

Trade-offs exist between recognition performance and computational efficiency.

Abstract

Speech Emotion Recognition (SER) plays a key role in advancing human-computer interaction. Attention mechanisms have become the dominant approach for modeling emotional speech due to their ability to capture long-range dependencies and emphasize salient information. However, standard self-attention suffers from quadratic computational and memory complexity, limiting its scalability. In this work, we present a systematic benchmark of optimized attention mechanisms for SER, including RetNet, LightNet, GSA, FoX, and KDA. Experiments on both MSP-Podcast benchmark versions show that while standard self-attention achieves the strongest recognition performance across test sets, efficient attention variants dramatically improve scalability, reducing inference latency and memory usage by up to an order of magnitude. These results highlight a critical trade-off between accuracy and efficiency, providing practical insights for designing scalable SER systems.