Mingling or Misalignment? Temporal Shift for Speech Emotion Recognition with Pre-trained Representations

TL;DR

This paper introduces a parameter-free temporal shift module to enhance speech emotion recognition models by better integrating information, leading to improved performance on the IEMOCAP benchmark.

Contribution

It proposes a novel temporal shift module for pre-trained speech models, improving emotion recognition without adding parameters or FLOPs.

Findings

Outperforms state-of-the-art on IEMOCAP dataset

Effective in both finetuning and feature extraction settings

Enhances model performance with no additional computational cost

Abstract

Fueled by recent advances of self-supervised models, pre-trained speech representations proved effective for the downstream speech emotion recognition (SER) task. Most prior works mainly focus on exploiting pre-trained representations and just adopt a linear head on top of the pre-trained model, neglecting the design of the downstream network. In this paper, we propose a temporal shift module to mingle channel-wise information without introducing any parameter or FLOP. With the temporal shift module, three designed baseline building blocks evolve into corresponding shift variants, i.e. ShiftCNN, ShiftLSTM, and Shiftformer. Moreover, to balance the trade-off between mingling and misalignment, we propose two technical strategies, placement of shift and proportion of shift. The family of temporal shift models all outperforms the state-of-the-art methods on the benchmark IEMOCAP dataset under both finetuning and feature extraction settings. Our code is available at https://github.com/ECNU-Cross-Innovation-Lab/ShiftSER.

Tables

Table 1. Table 1 : Comparison of UA with finetuning methods on IEMOCAP. Results are from original paper.

Type	Method	UA (%)
Baseline	CNN-BiLSTM	66.9
	Vanilla Finetuning	69.9
	P-TAPT	74.3
	TAP+HuBERT Large	74.2
Temporal Shift Augment	LSTM+Augment	74.1
	Transformer+Augment	74.8
	CNN+Augment	74.3
Temporal Shift	ShiftLSTM (ours)	74.5
	Shiftformer (ours)	74.7
	ShiftCNN (ours)	74.8

Table 2. Table 2 : Comparison with feature extraction on IEMOCAP. The results of pre-trained models are cited from [ 4 ] and others are from original paper. To be consistent with prior works, we adopt wav2vec 2.0 Base features.

Feature	Method	WA (%)	UA (%)
Convention	AR-GRU	66.9	68.3
	SeqCap with NAS	70.5	56.9
Fusion	eGeMAPS+wav2vec 2.0	-	66.3
	MFCC+Spec.+w2E	69.8	71.1
Pre-trained	wav2vec 2.0 Large	65.6	-
	HuBERT Large	67.6	-
	WavLM Large	70.6	-
	ShiftCNN (ours)	71.9	72.8
	ShiftLSTM (ours)	69.8	70.6
	Shiftformer (ours)	72.1	72.7

Table 3. Table 3 : Ablation study for components of building blocks.

Ablation	Variant	FeatEx.	Finetune
CNN	baseline	71.3	74.1
	conv$\to$depthwise	68.7	74.2
	LN$\to$BN	70.2	73.7
	residual$\to$shift	71.7	74.8
Transformer	baseline	69.5	74.5
	RPE$\to$APE	61.6	73.7
	residual$\to$shift	69.3	74.5
	MHSA$\to$pooling	69.5	73.7
	MHSA$\to$shift	70.3	74.7
LSTM	baseline	71.8	74.3
	identity$\to$shift	71.9	74.5

Full text

**Index Terms— ** Speech emotion recognition, wav2vec 2.0, HuBERT

1 Introduction

Speech emotion recognition (SER) refers to recognizing human emotion and affective states from audio. Benefiting from different emotional datasets annotated with categorical or dimensional labels, deep neural networks (DNNs) have shown great capability of recognizing emotion from speech efficiently [1]. The standard practice is training from scratch with conventional features [2]. However, the relatively small SER datasets [3], containing only a few speakers and utterances, and conventional hand-crafted acoustic features restrict the performance [4].†† This paper is funded by Special Project on Support for Regulative Intelligence Technology under the Shanghai Science and Technology Innovation Plan 2022 on ”Research on the Theory, Assessment System and Prototype System for Enhancing Emotional Cognition in the General Population” (Grant No. 22511105901).

⋆ Corresponding author. amzhou@cs.ecnu.edu.cn

Inspired by the success of pre-trained features like wav2vec 2.0 [5] and HuBERT [6] in other speech tasks, some researchers begin to validate its superiority over hand-engineered features in SER [7]. Some works present fusion methods of pre-trained and traditional features [8] while others explore task adaptive pre-training strategies for SER [9]. However, most of previous works mainly focus on exploiting pre-trained representations and just adopt linear head on top of the pre-trained model, neglecting the design of downstream network. The building of specialist models for speech downstream tasks is also necessary for pre-trained representations.

Therefore, we take initiatives to explore the architecture of the networks for SER on pre-trained representations and propose a parameter-free, FLOP-free temporal shift module to promote channel mingling. We first draw inspirations from the advancement of Transformers [10] as well as modern CNNs [11] and investigate the common network configurations to set strong baselines. Then, as shown in Figure 1, we introduce an efficient channel-wise network module, where partial channels are shifted along the temporal dimension by one frame to mingle the past feature to the present one. The core idea of our temporal shift is to introduce channel-wise information mingling. It is noteworthy that the features are shifted partially and thus endows the whole model with partial receptive field enlargement, different from the sufficient frame-level receptive field enlargement by stacking pretrained CNN or Transformer blocks. Moreover, our temporal shift serves as the network module similar to the shift operations in computer vision [12] rather than the common augmentation applications in audio processing [13]. However, such channel-wise partial shift seems to go against the characteristic of alignment in many speech tasks [14]. From the perspective of alignment, the information contained in the shifted channels becomes inaccessible for the original frame, indicating misalignment along with mingling. To balance the trade-off between mingling and misalignment, we propose two strategies for applying temporal shift, including proportion and placement. Proportion of shift is defined to adjust the ratio of shifted channels and the placement of shift can be unified into two manners (Figure 2). To the best of our knowledge, we are the first to propose channel-wise shift operation in audio processing and apply temporal shift to three mainstream building blocks. The family of shift models, including ShiftCNN, Shiftformer and ShiftLSTM, all outperform the state-of-the-art methods on the benchmark IEMOCAP dataset under both finetuning and feature extraction settings.

2 Methodology

In this section, we first illustrate the property of temporal shift and its application strategy. Then we specify the design of basic building blocks and the shift variants.

2.1 Temporal Shift

As motivated, our temporal shift is to introduce channel-wise information mingling to the standard building block. Part of the channels will be shifted to its neighbouring time stamp and the rest channels remain as shown in Figure 1. The unidirectional shift mingles the past feature to the current one while the bidirectional shift combines the future feature with the current one additionally.

Nonetheless, the partial shifted data leads to frame misalignment and the original information contained in the shifted channels is inaccessible for that time stamp. Specifically, for each time stamp, the un-shifted features are forced to adapt to the neighboring shifted channels while the shifted channels are moved aside from the position it used to be. To balance the trade-off between mingling and misalignment, we provide two strategies for temporal shift.

Proportion of shift. The proportion of shifted channels can be defined as the hyperparameter $\alpha$, adjusting the bargain between mingling and misalignment. Intuitively, if $\alpha$ is kept small enough, the negative effect can be negligible. If $\alpha$ is set large, the information mingling is promoted but the underlying misalignment may cause performance degradation. We will measure the variants with shift proportion in $\left\{1/2,1/4,1/8,1/16\right\}$.

Placement of shift. Exploring the proper location of temporal shift in the neural network is essential. As shown in Figure 2, The manner of applying temporal shift to the model can be unified in two ways, including residual shift and in-place shift. In-place shift (Figure 2(a)) forces the model to fit the shifted data, where the effects of both sides are great. Residual shift (Figure 2(b)) is placed on the branch of the network, maintaining both the aligned data and misaligned data. The shifted representations from the residual connection can serve as a complementary to the original representations, behaving like an ensemble [15].

In addition to technical strategies, the underlying misalignment can be remitted by the SER task from pre-trained representations itself. Specifically, the characteristic of SER task indicates its focus on invariant and discriminative representations rather than alignments between the input and target sequence in other speech tasks [14].

2.2 Specification of Temporal Shift Models

We insert temporal shift module into three types of popular building blocks in SER, including convolution neural network, recurrent neural network, and Transformer. The number of channels $C$ matches the representations of pre-trained wav2vec 2.0 Base and HuBERT Base. The number of blocks $B$ is set to ensure the number of parameters roughly the same (9.5M). It is worth noting that our temporal shift can be inserted flexibly and the proportion of shift can be adjusted as the hyperparameter. For uniformity, we adopt the same architecture with fixed proportion in our experiments to validate its effectiveness. The isotropic architectures are summarized as follows.

• •

ShiftCNN: $C=(768,3072,768),B=2,\alpha=1/16$

• •

Shiftformer: $C=(768,3072,768),B=2,\alpha=1/4$

• •

ShiftLSTM: $C=(768,1536),B=1,\alpha=1/4$

CNN. We draw inspirations from the advancement of ConvNext blocks [11] and time-channel separable convolution. This pure convolution neural network architecture described in Figure 3(a) can serve as a strong baseline. For its shift variant ShiftCNN with kernel size 7, we have the last block with residual shift for 1/16 proportion, which enables channel-aligned modeling capability and subtle channel-wise information fusion.

Transformer. We employ the standard Transformer with relative positional encoding (RPE) as shown in Figure 3(b). However, the standard Transformer with residual shift brings no more improvement (Table 3 residual$\rightarrow$shift of Transformer). Inspired by the recent literature [16], we further sidestep self-attention and view Transformer as a more general architecture, consisting of token mixer and MLP. The building of Shiftformer (Figure 3(c)), closer to MLP-like architecture, is based on the replacement of self-attention with temporal shift as token mixer, involving a more radical bidirectional shift manner. Our shiftformer enjoys effectiveness of the general Transformer architecture as well as the channel-wise mingling of temporal shift.

RNN. Bidirectional long short-term memory (LSTM) [17] is set to capture both past and future contexts. ShiftLSTM is constructed by straightforward in-place shift with moderate 1/4 shift proportion.

3 Experiments

In this section, we first observe the main properties of temporal shift module. Next we demonstrate the superiority of the family of shift models on the benchmark dataset IEMOCAP under both finetuning and feature extraction settings. Finally, we ablate key components of our proposed models.

3.1 Experimental Setup

Dataset. Interactive emotional dyadic motion capture database (IEMOCAP) [3] is a widely-used benchmark SER dataset with total length of about 12h, containing five sessions with one male and one female each. We adopt leave-one-session-out 5-fold cross-validation and four emotion categories, including 1636 happy utterances, 1084 sad utterances, 1103 angry utterances, and 1708 neutral utterances. Each sample is clipped to 7.5 seconds and the sampling rate is 16kHz.

Evaluation. We do comparisons under two configurations, feature extraction and finetuning. Feature extraction [8] is to evaluate the sole downstream model, where the frozen pre-trained model is taken as the feature extractor and the downstream models are trained in the supervised manner. And under the finetuning configuration, the parameters of both upstream pre-trained models (only the Transformer layers) and downstream models are updated during training.

The unweighted accuracy (UA) and weighted accuracy (WA) are used as evaluation metrics. UA is computed as the average over individual class accuracies while WA is the correct prediction over the total samples. The reported accuracy is the average of 5 folds results.

Implementation Details. We conduct experiments with two representative self-supervised pre-trained models, wav2vec 2.0 Base [5] and HuBERT Base [6]. We finetune the models using Adam optimizer with learning rate of $5\times 10^{-4}$. We use AdamW optimizer with cosine scheduler and decay rate of 0.1 for feature extraction. We train at batch size of 32 for 100 epochs with 5 epochs of linear warm-up.

3.2 Properties of shift variants

We conduct experiments on HuBERT feature extraction and observe the behavior of temporal shift. Since our temporal shift module provides channel-wise enlargement of receptive field, the effect is closely related to that of the baseline model. Figure 4 varies CNN kernel sizes to probe the potential of temporal shift, where residual shift brings more gain with small kernels (from 71.3% to 72.1% for size 7) than large ones. Residual shift also achieves better performance than in-place shift for all relatively small kernels. From similar perspective as [15], the residual shift makes the model an implicit ensemble composed of shift and origin modules, thus benefiting from both channel-aligned and channel-mingled information at no extra computation. However, the projection shortcut added to bidirectional LSTM in Figure 4 introduces extra parameters and puts residual shift at a disadvantage, leading to nearly 1% degradation. The accuracy of Shiftformer increases steadily with proportion of bidirectional shift in Figure 4, surpassing the standard Transformer by about 1% with 27% fewer parameters. This suggests high proportion with bidirectional manner improves temporal modeling for MLP-like architecture and meanwhile avoids high complexity of self-attention operation.

3.3 Comparison with State-of-the-Arts

Baselines. We compare our proposed models with existing state-of-the-art methods under both finetuning and feature extraction settings. Finetuning methods include vanilla finetuning, P-TAPT, TAPT [9], CNN-BiLSTM [7] and SNP, TAP [18]. Feature extraction methods include conventional feature (AR-GRU [2], SeqCap with NAS [1]), feature fusion (eGeMAPS+wav2vec 2.0 [8], MFCC+Spectrogram+W2E [19]) and pre-trained features (wav2vec 2.0 [5], HuBERT [6] and WavLM [4]). Moreover, we adopt temporal shift as data augmentation on hidden states for comparison, namely employed randomly just in the training stage.

Comparison on finetuning. As shown in Table 1, the family of temporal shift models outperforms the state-of-the-art finetuning methods. For the adaptive finetuning methods, TAPT requires another Hubert-like pre-training stage on IEMOCAP while TAP utilizes a more advanced pre-trained model HuBERT Large, but still falling behind ours trained solely on wav2vec 2.0 Base by 0.6 %. Notably, our temporal shift taken as data augmentation also shows competitive performance, signifying the channel-wise mingling mechanism and our strong baseline building blocks.

Comparison on feature extraction. Table 2 is split into multiple parts to include methods adopting different types of features. We follow the evaluation protocol as [8], namely adopting the trainable weighted sum of embeddings. Taking the wav2vec 2.0 Base as the feature extractor, our ShiftCNN and Shiftformer outperform all the other methods, even attaining better performance than one of the latest advanced pre-trained model WavLM Large by 1.5%.

3.4 Ablation Studies

We conduct ablation studies on CNN, Transformer, and LSTM respectively in Table 3 under both wav2vec 2.0 finetuning and HuBERT feature extraction (dubbed FeatEx.). UA is reported in the table. The advancement of the overall architecture is verified by the ablation of key components, covering types of normalization, convolution and position encoding. Interestingly, Transformer with residual shift fails to catch up with the standard one while our shiftformer, replacing multi-head self-attention (MHSA) with shift operation, outperforming the others with 3.6M fewer parameters. This indicates the benefit of reducing complexity and introducing channel-wise information mingling.

4 Conclusion

In this paper, we propose temporal shift module to introduce channel-wise mingling for downstream models. We hope our shift variants inspire future research on downstream model design and look forward further application of temporal shift.