Empirical Interpretation of the Relationship Between Speech Acoustic Context and Emotion Recognition

TL;DR

This paper investigates how acoustic context and phone boundaries influence speech emotion recognition, proposing an attention-based distributed approach that aligns with psycholinguistic theories and improves understanding across different speech datasets.

Contribution

It introduces an attention-based method that models emotion as a distributed over continuous time-windows, bridging psycholinguistic insights with computational SER models.

Findings

Distributed approach enhances cross-corpora SER performance

Attention vectors reveal overlapping distributions of phonemes and emotion

Using fundamental frequency improves emotion localization

Abstract

Speech emotion recognition (SER) is vital for obtaining emotional intelligence and understanding the contextual meaning of speech. Variations of consonant-vowel (CV) phonemic boundaries can enrich acoustic context with linguistic cues, which impacts SER. In practice, speech emotions are treated as single labels over an acoustic segment for a given time duration. However, phone boundaries within speech are not discrete events, therefore the perceived emotion state should also be distributed over potentially continuous time-windows. This research explores the implication of acoustic context and phone boundaries on local markers for SER using an attention-based approach. The benefits of using a distributed approach to speech emotion understanding are supported by the results of cross-corpora analysis experiments. Experiments where phones and words are mapped to the attention vectors along with the fundamental frequency to observe the overlapping distributions and thereby the relationship between acoustic context and emotion. This work aims to bridge psycholinguistic theory research with computational modelling for SER.