Speech Swin-Transformer: Exploring a Hierarchical Transformer with Shifted Windows for Speech Emotion Recognition

TL;DR

This paper introduces Speech Swin-Transformer, a hierarchical Transformer model with shifted windows designed to capture multi-scale emotional features in speech signals, significantly improving speech emotion recognition performance.

Contribution

The paper proposes a novel hierarchical speech Transformer with shifted windows that effectively aggregates multi-scale emotional features for speech emotion recognition.

Findings

Outperforms state-of-the-art SER methods

Effective multi-scale emotion feature aggregation

Demonstrates superior hierarchical speech representation

Abstract

Swin-Transformer has demonstrated remarkable success in computer vision by leveraging its hierarchical feature representation based on Transformer. In speech signals, emotional information is distributed across different scales of speech features, e.\,g., word, phrase, and utterance. Drawing above inspiration, this paper presents a hierarchical speech Transformer with shifted windows to aggregate multi-scale emotion features for speech emotion recognition (SER), called Speech Swin-Transformer. Specifically, we first divide the speech spectrogram into segment-level patches in the time domain, composed of multiple frame patches. These segment-level patches are then encoded using a stack of Swin blocks, in which a local window Transformer is utilized to explore local inter-frame emotional information across frame patches of each segment patch. After that, we also design a shifted window Transformer to compensate for patch correlations near the boundaries of segment patches. Finally, we employ a patch merging operation to aggregate segment-level emotional features for hierarchical speech representation by expanding the receptive field of Transformer from frame-level to segment-level. Experimental results demonstrate that our proposed Speech Swin-Transformer outperforms the state-of-the-art methods.