Pitch-Synchronous Single Frequency Filtering Spectrogram for Speech Emotion Recognition

TL;DR

This paper introduces a pitch-synchronous single frequency filtering spectrogram for speech emotion recognition, demonstrating improved accuracy over traditional methods by capturing both time and frequency resolutions effectively.

Contribution

It proposes a novel pitch-synchronous SFF spectrogram, enhancing speech representation for emotion recognition and outperforming state-of-the-art STFT-based CNN approaches.

Findings

Achieved 70.4% weighted accuracy on IEMOCAP.

Improved recognition of happy emotions to 22.7%.

Outperformed STFT spectrogram methods by +4.3% (weighted).

Abstract

Convolutional neural networks (CNN) are widely used for speech emotion recognition (SER). In such cases, the short time fourier transform (STFT) spectrogram is the most popular choice for representing speech, which is fed as input to the CNN. However, the uncertainty principles of the short-time Fourier transform prevent it from capturing time and frequency resolutions simultaneously. On the other hand, the recently proposed single frequency filtering (SFF) spectrogram promises to be a better alternative because it captures both time and frequency resolutions simultaneously. In this work, we explore the SFF spectrogram as an alternative representation of speech for SER. We have modified the SFF spectrogram by taking the average of the amplitudes of all the samples between two successive glottal closure instants (GCI) locations. The duration between two successive GCI locations gives the pitch, motivating us to name the modified SFF spectrogram as pitch-synchronous SFF spectrogram. The GCI locations were detected using zero frequency filtering approach. The proposed pitch-synchronous SFF spectrogram produced accuracy values of 63.95% (unweighted) and 70.4% (weighted) on the IEMOCAP dataset. These correspond to an improvement of +7.35% (unweighted) and +4.3% (weighted) over state-of-the-art result on the STFT sepctrogram using CNN. Specially, the proposed method recognized 22.7% of the happy emotion samples correctly, whereas this number was 0% for state-of-the-art results. These results also promise a much wider use of the proposed pitch-synchronous SFF spectrogram for other speech-based applications.

Tables

Table 1. TABLE I : Parameters of each CNN-Block

Name	Layers	Kernel Size	Output Size
CNN Block1	Convolution2D	12 $\times$ 16	189 $\times$ 284 $\times$ 16
	Max Pooling2D	100 $\times$ 150	90 $\times$ 135 $\times$ 16
CNN Block2	Convolution2D	8 $\times$ 12	83 $\times$ 124 $\times$ 24
	Max Pooling2D	50 $\times$ 75	34 $\times$ 50 $\times$ 24
CNN Block3	Convolution2D	5 $\times$ 7	30 $\times$ 44 $\times$ 32
	Max Pooling2D	25 $\times$ 37	6 $\times$ 8 $\times$ 32
Fully Connected Layer		64	64
Output Layer		4	4

Table 2. TABLE II : Comparison of Weighted accuracy and Unweighted accuracy of the proposed method with the baseline method

	Representation	UWA	WA
Proposed	Pitch-Synchronous SFF Spectrogram	63.95	70.4
	SFF-20 ms	59.09	67.84
Baseline	STFT-Spectrogram	58.55	67.04
Satt et al.[32]	STFT Spectrogram	56.6	66.1

Table 3. TABLE III : Confusion matrix of STFT spectrogram in percentage

Emotion	Anger	Happy	Neutral	Sad
Anger	91.67	0	0	8.33
Happy	40.91	0	50	9.1
Neutral	17.86	3.57	52.67	25.89
Sad	0	0	10.12	89.87

Table 4. TABLE IV : Confusion matrix for proposed pitch-synchronous SFF spectrogram in percentage

Emotion	Anger	Happy	Neutral	Sad
Anger	81.25	4.16	14.58	0
Happy	27.27	22.7	45.45	4.55
Neutral	6.25	9.82	64.28	19.64
Sad	0	0	13.92	86.07

Equations

$$p[n]=s[n]-s[n-1]$$

$$p[\bar{k},n]=p[n]e^{j\overline{{\omega }_k}n}$$

$$\overline{{\omega }_k}=\frac{2\pi \overline{f_k}}{f_s}$$

$$H(z)=\frac{1}{1+r{z}^{-1}}$$

$$y[k,n]=-ry[k,n-1]+p[\bar{k},n]$$

$$e[k,n]=\sqrt{y_r^2[k,n]+y_i^2[k,n]}$$

$$u[k,l]=\frac{1}{s[l+1]-s[l]}\sum _{i=s[l]}^{s[l+1]}e[k,i],\forall s[l]$$

$$x[k,n]=log(u[k,l])$$

$$p[n]=s[n]-s[n-1]$$

$$z_0[n]=-\sum _{i=1}^2{a}_i{z}_0[n-i]+p[n]$$

$$z[n]=z_0[n]-\sum _{m=-M}^M{z}_0[n+m]$$

$$O(m,n)$$

$$=\sum _{a=-x}^{a=+x}\sum _{b=-y}^{b=+y}s(a,b)\cdot w(m-a,n-b)$$

$$O_m^l=\sigma (BN(b_m^l+\sum _n{O}_n^{l-1}\ast w_{mn}^l))$$

$$\sigma (x)=\{\begin{array}{ll}x& x\ge 0\\ 0& x<0\end{array}$$

$$O^l=\underset{\forall pϵ\Omega }{\max }{O}_p^l$$

$$L_i=-\sum _{x=1}^C{i}_{x,c}\ast log(p_{x,c})$$

$$L=\frac{1}{N}\sum _{i=1}^N{L}_i$$

$$m_t={\beta }_1\ast m_{t-1}+(1-{\beta }_1)g_t$$

$$v_t={\beta }_2\ast v_{t-1}+(1-{\beta }_2){g_t}^2$$

$$\widehat{m_t}=\frac{m_t}{1-{\beta }_1^t}$$

$$\widehat{v_t}=\frac{v_t}{1-{\beta }_2^t}$$

$${\theta }_{t+1}={\theta }_t-\frac{\eta }{\sqrt{\widehat{v_t}}+ϵ}\widehat{m_t}$$

$$z_i=\sum _j{h}_j{W}_{ji}$$

$$\sigma (z{)}_i=\frac{e^{z_i}}{{\sum }_{j=1}^n{e}^{z_j}}$$

$$\widehat{y_t}=\underset{i}{\mathrm{argmax}}{p}_i$$

$$s=\underset{n=\mathrm{1...}N}{\mathrm{argmax}}\frac{{\sum }_{t=1}^{T}p(y_t|x)}{T}$$

Full text

I Introduction

Speech emotion recognition (SER) refers to the classification/recognition of the person’s emotional state using the speech signal. SER has a lot of applications in real life. It can be beneficial for applications where natural human-computer interaction is required. In computer tutorial applications, the detected emotion of the user can help the system in responding to the user’s query [11]. It can be incorporated in the onboard system of a car for initiating the safety of a passenger depending on his mental state [33]. Medical professionals can use it as a diagnostic tool in psychological treatment [13]. In automatic translation systems, it can help in effectively conveying the emotions between two parties[2].

For recognizing emotions from a speech, we need to extract emotion specific features, which are invariant with text[37]. After the introduction of deep neural networks, SER has gone through a significant growth over the past few years. There are two ways for extracting features from speech: (i) hand-crafted features and (ii) deep neural network based features. Popular hand-crafted features are formant locations/bandwidths, pitch, voice probability, zero-crossing rate, harmonics-to-noise ratio, mel filter bank features, mel frequency cepstral coefficients (MFCCs), energy, and jitter [22, 24, 34, 1]. There are two disadvantages with hand crafted features. First is the inherent inaccuracies in extraction of these features. Second, the handcrafted features tend to overlook the higher level features that can be derived from the lower level features. On the other hand, in deep learning features are learned in a hierarchical manner; learning higher level abstractions of the low-level features.

Recently convolution neural networks (CNN) have become very popular for SER. CNN is a data-driven feature extractor in which filters are learned using data itself [28, 23, 35, 5, 6]. Short time Fourier transform (STFT) spectrogram is widely used as an input to the CNN for speech applications [23, 32, 5, 12]. Spectrogram is a time-frequency representation of a signal. There are various existing works in SER [23, 32, 5, 12] in which spectrogram was given as input to a CNN. In [23] the spectrogram was used as input to a CNN with sparse auto-encoder to find salient features for SER. In [5] experiments were conducted using pre-trained AlexNet model and a freshly trained CNN model. The pre-trained AlexNet model was fine-tuned for transfer learning. The experimental results showed that the proposed approach based on the freshly trained CNN model were better than the pre-trained AlexNet model. In [12] experiments were conducted on several CNN and LSTM-RNN architectures. The CNN architectures achieved better performance as compared to the other LSTM-RNN architectures.

Reference [32] highlights the drawback of spectrogram representation for noisy data. The authors modified the spectrogram using pitch information for robust SER. They also commented on Mel-scale spectrograms vs. linear-spaced spectrogram. Mel-scale spectrograms remove the pitch information but the emotions are strongly correlated with the pitch information. Therefore, linear-spaced spectrogram is usually used for speech emotion recognition. In [40] spectrogram along with phoneme embedding features are combined with a CNN model to retain emotional contents of speech. Both spectrogram and hand-crafted features have been used as input to the CNN for SER. However, as noted in [25], the raw spectrogram has higher accuracy than the hand crafted features because the latter are already decorrelated.

Speech signal is a non-stationary signal, i.e., it’s behaviour changes with time. It is processed frame-wise, where the size of a frame is small enough (typically around 25 ms) to assume that the characteristics of speech is stationary within a frame. In spectrogram, Fourier transform of a short-time windowed signal is taken with some overlapping (typically around 10 ms). The shorter sized window (wide-band spectrogram) contains high temporal resolution but less spectral resolution, whereas the larger sized window (narrow-band spectrogram) contains high spectral resolution but less temporal resolution [7]. In other words, STFT spectrogram is unable to obtain high resolutions of time and frequency simultaneously. Since, the pattern of emotional speech varies rapidly within a glottal cycle due to the opening and closing of the glottis [17, 38], it requires high temporal resolution for further analysis.

Wavelet transform [16] is another time frequency representation that overcomes the limitation of the STFT spectrogram to some extent. Here, the time resolution is better in the high frequency region, while frequency resolution is better in the low frequency region. However, in wavelet transform, the type of wavelets have to be chosen a priori, which often leads to misinterpretation of the data. On the other hand, the newly proposed single frequency filtering (SFF) spectrogram [3, 4, 29, 17] has both high temporal and spectral resolution without any requirement of a priori choices. Further, in absence of sensitive parameters like window-size (as in the case of STFT), SFF is more robust. SFF technique has been successfully used in speech/non-speech detection [3], fundamental frequency extraction [4, 29] and glottal closure instants (GCIs) detection [17]. To the best of our knowledge SFF has not been used for SER.

SFF spectrogram is especially appealing for SER because the behaviour of emotional speech is more non-stationary than normal speech. SFF output contains both high temporal and spectral resolutions. Hence, the SFF spectrogram captures transient parts of the emotional speech very well. The high spectral resolution clearly represents harmonic structures. The harmonics patterns, called timbre spectrum, are important for emotion classification [20, 14]. However, there is a practical problem in using SFF spectrogram for SER. SFF spectrogram extracts the amplitude envelope at each sampling instant. This leads to a huge feature matrix, making it computationally prohibitive to be used as the input to a CNN. Hence, a logical way is required to reduce the size of the feature matirx of SFF spectrogram.

Based on the above discussion, there are two primary motivations of our work:

• •

Use of SFF Spectrogram for SER. The advantages offered by the SFF spectrogram over the STFT spectrogram and the wavelet transform technique make a strong case for exploring SFF spectrogram for SER.

• •

Reduction of the size of the feature matrix of SFF. A logical way is required to reduce the size of the feature matirx of SFF spectrogram to make its use computationally practical.

In light of the above motivations, the contributions of our work are as follows:

• •

Pitch-synchronous SFF spectrogram: We have proposed a novel pitch-synchronous SFF spectrogram, where the amplitude is averaged over pitch cycles. As compared to block processing – where the assumption that the emotional speech is stationary over fixed-size frames does not hold good practically – the proposed approach logically decomposes frames based on GCI locations; an emotional speech is near stationary between two successive GCI locations. As a result, the unique pitch pattern of every emotion is reflected well in the pitch-synchronous SFF spectrogram (details in Sec. IV). To overcome the problem of a huge feature matrix, we have averaged the amplitudes over pitch cycles, hence, reducing the size of the feature matrix. This also motivated us to name the modified SFF spectrogram as pitch-synchronous SFF spectrogram.

• •

Improved performance: We have compared pitch-synchronous SFF spectrogram with (i) STFT spectrogram and (ii) SFF spectrogram with 20 ms fixed sized frames (called “SFF-20 ms spectrogram”). All representations were evaluated on a developed CNN model. The SFF-20 ms spectrogram achieves an improvement of 2.49% for unweighted accuracy and 1.74% for weighted accuracy over state-of-the-art STFT spectrogram using CNN [32]. Pitch-synchronous SFF spectrogram further improves upon this by achieving improvements of 7.35% for unweighted accuracy and 4.3% for weighted accuracy over state-of-the-art STFT spectrogram using CNN [32].

The rest of the paper is structured as follows. In Section @slowromancapii@, the proposed pitch-synchronous SFF spectrogram and CNN architecture are discussed in detail. Section @slowromancapiii@ describes experimental setup. Results are discussed in Sec. @slowromancapiv@. Section @slowromancapv@ concludes the paper with future directions.

II Proposed Approach

Our proposed work utilizes the SFF spectrogram [3, 4, 29, 17], which is a time-frequency representation, for SER. The SFF spectrogram derives the amplitude envelope of a signal at each frequency as a function of time. In this way, the SFF spectrogram captures the temporal variation at each sample in an emotional speech. Note that this variation can also be captured by the discrete Fourier transform (DFT) over a block of data at every sampling instant. But, this process is computationally expensive since the DFT is performed at each sample. The SFF spectrogram extracts the amplitude envelope at each sampling instant; resulting in a huge feature matrix. Computationally, it is impractical to use such a huge feature matrix as an input to CNNs. To overcome this, the amplitude envelope is averaged for all the samples between two successive GCI locations (called the pitch period). The resultant output representation is called pitch-synchronous SFF spectrogram. In section II-A, the detailed steps of constructing the pitch-synchronous SFF spectrogram are described. In Sec. II-B glottal closure instants (GCI) detection using zero frequency filtering (ZFF) method is described. In Sec. II-C the proposed deep CNN architecture is described.

II-A Pitch synchronous SFF spectrogram

Following are the steps to obtain the SFF spectrogram output.

The speech signal $s[n]$, sampled at frequency $f_{s}$ Hz, is pre-emphasized ($p[n]$) to remove the low frequency bias from the speech signal.

[TABLE] 2. 2.

The pre-emphasized speech signal $p[n]$ is multiplied by a complex sinusoid of normalized shifted frequency as follows:

[TABLE]

where,

$p[\bar{k},n]$ is the resultant output at $n^{th}$ sample and

$\bar{\omega_{k}}$ is the normalized version of frequency $f_{k}$ at $k^{th}$ filter. $\bar{\omega_{k}}$ is computed as:

[TABLE]

where,

$\bar{f_{k}}$ is the shifted frequency shifted as $\bar{f_{k}}=f_{s}/2-f_{k}$.

In the above, $n\in{1...N}$ and $k\in{1...K}$, where $N$ is the total number of samples and $K$ is the total number of filters. If the spacing between the filters is $\Delta f$ Hz, the total number of filters are $K=\frac{f_{s}/2}{\Delta f}$. 3. 3.

The resultant output $p[\bar{k},n]$ is passed through a single pole filter at $f_{s}/2$ whose transfer function is given by

[TABLE]

The stability of the filter is ensured by choosing the value of the pole of the filter inside the unit circle (radius $r=0.9394)$. The output of SFF gives high resolution at each frequency because it considers the effect of one resonator at a time and reduces the effect of other resonators significantly. 4. 4.

The output of the filter is given by

[TABLE]

$y[k,n]$ is a complex number with real part $y_{r}[k,n]$ and imaginary part $y_{i}[k,n]$. 5. 5.

The SFF envelope, denoted $e[k,n]$, of the filtered output $y[k,n]$ at the $k^{th}$ filter is given by

[TABLE]

In out work, the SFF envelope is calculated for frequency range $[0,f_{s}/2]$ Hz with $20$ Hz spacing. The SFF envelope of an anger utterance is shown in Fig 1. 6. 6.

The GCI locations from the speech signal $s[n]$ are detected using the ZFF algorithm [26] described in Sec. II-B. Let $z[n]$ denote the ZFF signal and $s[l]$ denote the set of GCI locations extracted from $z[n]$. 7. 7.

Subsampling: It refers to the process of reducing the number of samples by averaging the samples between two successive GCI locations in the SFF envelope. The average values are computed as:

[TABLE]

where $s[l]$ is the GCI location and $i$ iterates over the samples between successive GCI locations $s[l]$ and $s[{l+1}]$. The resultant output is called pitch-synchronous SFF envelope.

Finally, the log-spectrum of the pitch-synchronous SFF envelope is computed as follows:

[TABLE]

The flow diagram of the proposed pitch-synchronous SFF spectrogram is shown in Fig 2.

II-B * GCI detection using the Zero Frequency Filtering (ZFF) method*

The GCI locations are detected by the ZFF method proposed in [26]. Following are the steps to detect the GCIs using the ZFF method.

Speech signal is differentiated to pre-emphasize the high frequency components as:

[TABLE]

The resultant signal is denoted as $p[n]$ and known as pre-emphasized signal. 2. 2.

The pre-emphasized signal $p[n]$ is passed through a zero frequency resonator. The response of the zero frequency resonator is defined as:

[TABLE]

where $z_{0}[n]$ is the resonator output, $a_{1}=-2$ and $a_{2}=1$. 3. 3.

For removing the trend in the resonator output $z_{0}[n]$ a moving average filter is used whose window length corresponds to the pitch period of that utterance. The resulting output is called the ZFF signal:

[TABLE]

where $2M+1$ corresponds to the number of samples used for computing the trend. 4. 4.

The positive zero crossing of the ZFF signal $z[n]$ corresponds to the epoch location and is denoted by $s[l]$.

Figure 3(a) shows the voiced segment of a sentence . Fig. 3(b) shows the differentiated eloctro-glottograph (DEGG) signal of the voiced segment of 3(a). Figure 3(c) shows the zero frequency filtered (ZFF) signal and Fig. 3(d) shows the positive crossing of the ZFF signal which corresponds to the epoch location.

II-C Deep CNN Architecture

Deep learning helps in learning multiple levels of abstraction by composing networks into multiple layers. The network learns features in a hierarchical manner. Low-level features are learned from the raw data and higher level features are learned from the lower level features. Thus, deep learning overcomes the dependency of shallow networks on hand-crafted features by learning the features itself. In general, a convolutional neural network (CNN) is composed of convolution layers and pooling layers. A convolution layer learns the filters for low-level features by itself, whereas traditionally, these filters were designed by human experts. Due to this advantage of automatically learning the features, raw inputs are directly fed to a CNN and it extracts the features from the input. The max-pooling layers extract dominant features that are robust against distortion and noise. Additional convolution and max-pooling layers are capable of deriving more complex features from intermediate features. A CNN is composed of repetitive units called CNN blocks. In this section the architecture of the CNN block used in our work and the learning procedures are discussed.

II-C1 CNN block

The CNN block used in our work consists of four components: one convolution layer, one batch normalization (BN) layer [15], one rectified exponential linear unit (ReLU) layer [27], and one max-pooling layer, in that order, as shown in Fig. 4. The core layers of the CNN block are convolution layer and pooling layer. The BN layer helps in improving the stability and performance of deep neural networks. It normalizes the activations at each batch and maintains zero mean and unit variance [15]. ReLU layer uses ReLU activation function. It is widely used because it can lead to higher recognition accuracies and faster convergence rates [27]. Max-pooling layer uses a max-filter that is applied to the sub-regions of the output of the convolution layer [9]. Depending upon the requirements, these CNN blocks can be configured accordingly.

In a convolution layer, each index value $s(m,n)$ of the input $s$, which is a 2D matrix, is convolved with the convolution kernels. The input for the first layer is the raw spectrogram. The input for the convolution layers in the subsequent CNN blocks is the output of the previous CNN block. The convolution operation is performed between the convolution kernel and the input data to produce the feature maps. The convolution kernel $w(m,n)$ of size $(x,y)$ is initialized randomly. When $s(m,n)$ is convolved with convolution kernel $w(m,n)$, the resultant output $O(m,n)$ is obtained as:

[TABLE]

The resultant output is given as input to the BN layer. The features that are learned by a convolution layer are normalized by the corresponding BN layer. The output of a BN layer is given to the corresponding ReLU layer as follows:

[TABLE]

where,

$O_{m}^{l}$ represents the $m^{th}$ output feature at the $l^{th}$ layer,

$O_{n}^{l-1}$ represents the $n^{th}$ input feature at the $(l-1)^{th}$ layer, and

$w_{mn}^{l}$ denotes the convolution kernel between the $m^{th}$ and $n^{th}$ feature.

The ReLU activation function $\sigma$(.) of the network is defined as follows:

[TABLE]

where $x$ denotes the activation potential.

The next layer is max-pooling or downsampling layer. The normalized features are passed to this layer. The resolution of the features is reduced by taking the maximum value of a sub-region in the pooling layer. The output of a max-pooling layer can be defined as:

[TABLE]

where,

$O^{l}$ represents the output features,

$O_{p}^{l}$ represents the input features, and

$\Omega$ depicts the pooling patch.

II-C2 Learning Procedures

Learning of the neural network is phrased as an optimization problem to minimize the loss between the targeted and the predicted output. Our work is formulated as a multi-class classification problem. Therefore, the categorical cross-entropy loss[10] function is used. The cross-entropy loss gives a probability value between zero and one to measure the performance of the corresponding classification model. As the predicted probability diverges from the target value, the cross-entropy loss increases. A separate loss for each class label per observation $i$ is calculated. All these losses are summed and the combined loss is obtained as follows:

[TABLE]

where,

$C$ is the number of classes,

$i$ is a binary indicator; it is one for correct classification and zero otherwise, and

$p_{x,c}$ is the predicted probability of sample $x$ for class $c$.

The cross-entropy loss over a mini-batch is computed by taking the average:

[TABLE]

where $N$ is the number of samples in a batch.

For computing the gradients, the cost function is differentiated with respect to the model parameters and backpropagated to prior layers using the backpropagation algorithm [21, 31]. After obtaining the gradients, in general, the gradient descent algorithm or one of it’s variants is used for updating the model parameters. In our work, Adam optimizer [19, 30] has been used for loss optimization. Adam stands for adaptive moment estimation. Using Adam, for each parameter, adaptive learning rates are computed. In addition to computing an exponentially decaying weighted average of past gradients $m_{t}$, similar to momentum, Adam also computes an exponentially decaying weighted average of the squared gradients $v_{t}$, where $\beta_{1}$ and $\beta_{2}$ control the decaying rates of these moving averages as shown below:

[TABLE]

[TABLE]

where $g_{t}$ denotes the gradient value at step $t$. The initial value of the moving averages $\beta_{1}$ and $\beta_{2}$ is close to one. This results in a bias of moment estimates towards zero. To counteract this, a bias correction is applied as follows:

[TABLE]

[TABLE]

where,

$m_{t}$ is the first momentum representing the mean of gradients and

$v_{t}$ is the second momentum representing the uncentered variance of gradients.

For updating the parameters, the final formula is given below:

[TABLE]

where,

$\theta$ is the parameter and

$\eta$ is the learning rate.

The classifier used in this architecture is softmax. Softmax is a generalization of logistic regression for multi-class classification. The softmax classifier gives the probabilities for each class label. The softmax function is defined as:

[TABLE]

[TABLE]

where,

$z_{i}$ is the input to softmax,

$W_{ji}$ is the weight between the penultimate layer and softmax layer, and

$h_{j}$ is the activation in the penultimate layer.

The class label of each segmented utterance $\hat{y_{t}}$ is predicted as follows:

[TABLE]

where $p_{i}$ is the probability for class $i$.

III Experimental Setup

Our proposed model has been evaluated on the IEMOCAP (Interactive emotional dyadic motion capture)[8] dataset. The IEMOCAP dataset is a multi-modal dataset which contains audio, video, text, and gesture information of conversations arranged in dyadic sessions. In this work, we have considered only the audio tracks of the IEMOCAP dataset. The IEMOCAP dataset is a natural-like improvised dataset that reasonably resembles spontaneous real life emotional speeches.

The IEMOCAP dataset comprises of five sessions. In each session, there are two different speakers. There is no overlapping between speakers of different sessions. In each session, there is one male and one female. The conversation of one session is approximately five minutes long. The contents of the dataset are recorded in both scripted and spontaneous scenarios. In our study, we have considered four categorical (class) labelled emotions namely angry, happy, sad and neutral of improvised sessions[8]. We have used leave-one-speaker-out (LOSO) cross validation strategy. In LOSO four sessions are used for training. One speaker form the remaining fifth session is used for validation while the the other speaker is used for testing. This step is repeated for all the five sessions.

In this dataset, all utterances are of different lengths; varying from one to twelve seconds approximately. To deal with the variance in the lengths of the input utterances, the utterances are split into three second segments. Except the last segment, all segmented utterances are $T=3$ seconds long. Each segment is assigned the label of its parent utterance. These segmented utterances are used for training and validation. During testing, to retrieve the label at the utterance level, the posterior probabilities of all the constituent segments is averaged. The class label with the highest average value is assigned as the corresponding utterance label:

[TABLE]

where,

$s$ is the predicted label of the utterance,

$p(y_{t}|x)$ is the posterior probability of segment $t$,

$T$ is the number of segments, and

$N$ is the number of classes, i.e., four.

The pitch-synchronous SFF spectrograms are computed as described in Sec. II-A. There are only two parameters that need to be set: the value of the radius of the filter $r=0.9394$ and the spacing frequencies $\Delta f=20Hz$. The range of selected frequencies is $0-4kHz$, which covers human speech. The GCI locations are identified as described in Sec. II-B. The maximum number of GCI locations found in our experiments in a $T=3$ second segment is $1077$. Therefore, the size of feature matrix for a segment is $(200\times 1077)$, where $200$ is the number of frequency bins. The feature matrix of segments having less than $1077$ GCI locations is padded with zeros to reach the fixed size of $(200\times 1077)$. We compared the pitch-synchronous SFF spectrogram with the STFT spectrogram and the SFF- $20$ ms spectrogram. In the STFT spectrogram, the speech signal is segmented into frames of $40$ ms with an overlapping of $10$ ms. Hamming window is applied on each frame to compensate the end effect of the frame. The discrete Fourier transform of each frame is calculated with the window length of 800 (for $20$ Hz). In SFF-$20$ ms, the averaging of the amplitude of all the samples is performed with $50$% overlapping.

We used a convolutional neural network (CNN) for evaluating our proposed pitch-synchronous SFF. To ensure exactly the same experimental conditions, we have implemented the same CNN architecture as proposed in state-of-the-art method [32], except that we have used batch normalization for each convolution layer. The detailed parameters of each CNN block is given in Table I. The used $2D$ CNN network has the following structure. It has three CNN blocks (CNN Block1, CNN Block2, and CNN Block3), one fully connected layer and the output layer with the softmax classifier. The convolution and pooling kernels in each CNN block are $2-D$. In CNN Block1, there are $16$ convolution kernels of size $12\times 16$. CNN Block2 consists of $24$ kernels of $8\times 12$ size and CNN Block3 has $32$ kernels of size $5\times 7$. The sizes of the max-pooling layer of the first, second and third CNN blocks is $100\times 150$, $50\times 75$ and $25\times 37$ respectively.

The output obtained from the convolution layers represents high-level features. The output of the last CNN block is passed through the fully connected layer to learn non-linear combinations of these features for SER. The pitch-synchronous SFF spectrogram and the STFT spectrogram were implemented in MATLAB. The model is implemented using Keras, which is a library widely used for deep learning.

IV Result Discussion

The IEMOCAP dataset used in this work is significantly imbalanced. To cope with the imbalance issue, weights are assigned to each class during training such that the weight of a class is inversely proportional to the total number of samples in that particular class. The testing data is also imbalanced. Hence, we calculate unweighted as well as weighted accuracy. Unweighted accuracy is important for imbalanced data because it gives equal weightage to each class. Specifically, it is not biased against minority classes. Formally:

• •

Overall accuracy or weighted accuracy (WA) is defined as the number of samples predicted correctly out of the total number of samples.

• •

Average class accuracy or unweighted accuracy (UWA) is defined as the average accuracy of individual classes. The individual class accuracy is defined as the number of samples predicted correctly out of the total number of samples in that class.

The comparison of the pitch-synchronous SFF spectrogram with the STFT spectrogram for the four considered emotions is shown in Fig. 5. All the spectrograms are drawn from the same sentence of the same speaker. Pitch-synchronous SFF is an adaptive framing approach in which the frame-size is derived from the instantaneous pitch-period in the corresponding utterance. It can be seen that the pitch-synchronous SFF spectrogram captures clearer harmonics than the STFT spectrogram. The pitch-synchronous SFF spectrogram also contains more fine grained temporal information than the STFT spectrogram. These properties of the pitch-synchronous SFF spectrogram enable better discrimination between anger and happy emotions, which are known to produce very similar characteristics with respect to the existing features. The pitch-synchronous SFF spectrograms of happy and anger emotions exhibit clear energy and formant structures than the neutral emotion. Figure 5 (c) & (d) show the STFT spectrogram and the pitch-synchronous SFF spectrogram of neutral and sad emotions. The STFT spectrogram produces similar pattern for neutral and happy emotions, i.e., the energy concentrations are same across the entire frequency region. This is not good for discriminating among these two emotions. However, the pitch patterns of happy emotion are prominently different than the neutral emotion in the case of the pitch-synchronous SFF spectrograms. This results in a superior discrimination of happy and neutral emotions.

In the last layers of CNN architecture, the relationship between the feature maps are abstracted and the meaningful timbre spectrum is obtained. The GCI locations are present in the voiced regions. A frame is constructed based on the GCI locations, which automatically reduces the effect of silence region. We can clearly see in all the pitch-synchronous SFF spectrograms that the effect of silence between words is highly reduced.

Regularization techniques have been used to prevent overfitting. Regularization makes slight changes in the learning algorithms such that the model can generalize better, which improves the performance of the model on unseen data. This also helps in faster convergence. We have used multiple regularization techniques, namely batch normalization, dropout, LOSO cross validation, model selection, and early stopping. Batch normalization is used to normalize the convolution layer outputs to reduce the vanishing gradient problem in the activation layer. Batch normalization reduces the dependency of the designed network on weight initialization. It improves the gradient flow through the network and also adds some amount of regularization into the network since the empirical means and variances are calculated using samples from mini-batches. Therefore, batch normalization helps to increase the accuracy during testing.

Dropout has also been used as a regularization technique in this work. At each iteration, a random number of nodes are selected and removed. Thus, different number of nodes at every iteration result in different outputs. It is similar to an ensemble technique used in machine learning that captures more randomness. LOSO cross-validation is also used to avoid overfitting of the model. Experimental results show that the learned network is better generalized after cross-validation. The model was saved on the basis of maximum validation accuracy. The early stopping criteria is used to stop the model based on the value of the patience parameter. Patience is the number of epochs for which a model should continue training in spite of no improvement in the validation accuracy. The value of patience is chosen as five. In our experiments, several methods for reducing overfitting were used but overfitting was not completely avoided.

We have compared the proposed pitch-synchronous SFF spectrogram model with the following:

• •

State-of-the-art result on STFT spectrogram using CNN as proposed by Satt, Rozenberg and Hooory [32].

• •

The batch normalized model of state-of-the-art result [32] on the STFT spectrogram. We call this baseline model because any improvement of the proposed model over baseline is purely due to the pitch-synchronous SFF technique.

• •

SFF-20 ms spectrogram. This represents the use of the SFF spectrogram with the traditional block processing technique.

Table II shows the corresponding comparison results. The used regularization techniques improved WA and UWA by $+1.95$% and $+.94$% respectively over state-of-the-art [32]. There is further, albeit small, improvement when using the SFF-20ms spectrogram. The best results are obtained for the proposed pitch-synchronous SFF spectrogram with accuracy values of 63.95% (UWA) and 70.4% (WA). These correspond to improvements of 7.35% (UWA) and 4.3% (WA) over the state-of-the-art STFT spectrogram [32]. Looking at the results for the proposed pitch-synchronous SFF spectrogram and the SFF-20ms spectrogram, it can be said that a significant portion of improvement over state-of-the-art STFT spectrogram is due to logical windowing between two successive GCI locations, however, it must be noted that such logical windowing is not possible with the STFT spectrogram. This is because the STFT spectrogram does not obtain information at each instant of time.

The confusion matrices of the baseline STFT spectrogram and the pitch-synchronous SFF spectrogram are shown in Tables III and IV respectively. The accuracy of anger and sad emotions are satisfactory using the baseline STFT spectrogram. But, it fails to categorize happy emotion. All the samples of happy emotion are predicted as neutral using the STFT spectrogram. The same is also true for state-of-the-art results in [32] (the corresponding confusion matrix has not been shown here). However, the pitch-synchronous SFF spectrogram categorizes $22.7$% of happy emotions correctly. The accuracy of happy and neutral emotions are improved by $+22.7$% and $+11.61$% respectively in the case of the pitch-synchronous SFF spectrogram. This can be considered as a big plus for the the pitch-synchronous SFF spectrogram when compared to the STFT spectrogram. The accuracies of sad and anger emotions are approximately same for the STFT and pitch-synchronous SFF spectrograms.

The class accuracies of each emotion for the STFT, SFF-20 ms and pitch-synchronous SFF spectrograms are shown in Fig. 6. The SFF-20 ms spectrogram is computed from the SFF envelope. The framing procedure is same as the traditional STFT spectrogram where 20 ms frame size with 50% overlapping of previous frame is taken. The average of all the samples in a frame of an SFF envelope is computed. Using the SFF-20 ms spectrogram, an improvement of 2.49% (UWA) and 1.74% (WA) was observed as compared to the state-of-the-art STFT spectrogram using CNN [32]. The corresponding improvement for pitch-synchronous SFF spectrogram is 7.35% (UWA) and 4.3% (WA).

V Conclusion and Future Work

This paper highlights the drawbacks of feature representation of the existing STFT spectrogram for SER. We have proposed a novel pitch-synchronous SFF spectrogram for SER that overcomes these drawbacks. We have attempted to solve the problem of SER using deep convolutional neural networks. Our proposed architecture consists of three CNN blocks followed by a fully connected layer and an output layer for detecting four emotions (i.e. Anger, Happy, Neutral, Sad). On IEMOCAP dataset, the proposed pitch-synchronous SFF spectrogram achieved an improvement of $+7.35$% and $+4.3$% for weighted and unweighted accuracy values respectively over state-of-the-art STFT spectrogram representation using CNN. Specially, the proposed pitch-synchronous SFF spectrogram recognized 22.7% of the happy emotion samples correctly, whereas this number was 0% for state-of-the-art STFT spectrogram representation using CNN.

Pitch-synchronous SFF spectrogram can be used in various other applications such as speaker identification [18, 36], speaker verification [39], audio classification [41] etc. The SFF output has high SNR values of speech in the time-frequency domain. Our future work is to explore these properties of SFF to develop a robust speech emotion recognition against degradations due to noise.

ACKNOWLEDGEMENTS

Akshay Deepak has been awarded Young Faculty Research Fellowship (YFRF) of Visvesvaraya PhD Programme of Ministry of Electronics & Information Technology, MeitY, Government of India. In this regard, he would like to acknowledge that this publication is an outcome of the R&D work undertaken in the project under the Visvesvaraya PhD Scheme of Ministry of Electronics & Information Technol- ogy, Government of India, being implemented by Digital India Corporation (formerly Media Lab Asia).