End-to-End Multimodal Emotion Recognition using Deep Neural Networks

TL;DR

This paper presents an end-to-end deep neural network system that combines auditory and visual data for improved spontaneous emotion recognition, outperforming traditional handcrafted feature methods on the RECOLA dataset.

Contribution

It introduces a multimodal emotion recognition framework using CNN, ResNet, and LSTM networks trained jointly for the first time in an end-to-end manner.

Findings

Significantly outperforms traditional handcrafted feature approaches.

Effective integration of speech and visual modalities enhances emotion prediction.

Demonstrates robustness in recognizing natural emotions in spontaneous settings.

Abstract

Automatic affect recognition is a challenging task due to the various modalities emotions can be expressed with. Applications can be found in many domains including multimedia retrieval and human computer interaction. In recent years, deep neural networks have been used with great success in determining emotional states. Inspired by this success, we propose an emotion recognition system using auditory and visual modalities. To capture the emotional content for various styles of speaking, robust features need to be extracted. To this purpose, we utilize a Convolutional Neural Network (CNN) to extract features from the speech, while for the visual modality a deep residual network (ResNet) of 50 layers. In addition to the importance of feature extraction, a machine learning algorithm needs also to be insensitive to outliers while being able to model the context. To tackle this problem, Long Short-Term Memory (LSTM) networks are utilized. The system is then trained in an end-to-end fashion where - by also taking advantage of the correlations of the each of the streams - we manage to significantly outperform the traditional approaches based on auditory and visual handcrafted features for the prediction of spontaneous and natural emotions on the RECOLA database of the AVEC 2016 research challenge on emotion recognition.

Tables

Table 1. Table I : The replication of each bottleneck architecture of the ResNet-50 along with the size of the features maps of the convolutions.

Bottleneck layer	Replication	Number of feature maps (1x1, 3x3, 1x1)
1st	3	64, 64, 256
2nd	4	128, 128, 512
3rd	6	256, 256, 1024
4th	3	512, 512, 2048

Table 2. Table II : Results (in terms of ρ c subscript 𝜌 𝑐 \rho_{c} ) on arousal and valence after 60 epochs when varying sequence length for speech and visual networks.

Sequence length	Arousal	Valence
Visual network
75	.293	.276
150	.363	.488
300	.193	.496
Speech network
75	.727	.345
150	.744	.369
300	.685	.130

Table 3. Table III : RECOLA dataset results (in terms of ρ c subscript 𝜌 𝑐 \rho_{c} ) for prediction of arousal and valence. In parenthesis are the performance obtained on the development set. In a) we optimised the models wrt. MSE whereas in b) wrt. ρ c subscript 𝜌 𝑐 \rho_{c} .

Predictor	Features	Arousal	Valence
a. Mean squared error objective
SVR	eGeMAPS	.318 (.489)	.169 (.210)
SVR	ComParE	.366 (.491)	.180 (.178)
BLSTM	eGeMAPS	.300 (.404)	.192 (.187)
BLSTM	ComParE	.132 (.221)	.117 (.152)
Proposed	raw signal	.684 (.728)	.249 (.312)
b. Concordance correlation coefficient objective
BLSTM	eGeMAPS	.316 (.445)	.195 (.190)
BLSTM	ComParE	.382 (.478)	.187 (.246)
Proposed	raw signal	.699 (.752)	.311 (.406)

Table 4. Table IV : RECOLA dataset results (in terms of ρ c subscript 𝜌 𝑐 \rho_{c} ) for prediction of arousal and valence. In parenthesis are the performance obtained on the development set. A dash is inserted if the results were not reported in the original papers.

Predictor	Features	Arousal	Valence
Baseline [29]	eGeMAPS	.648 (.796)	.375 (.455)
RVM [30]	eGeMAPS	- (.750)	- (.396)
Audio BN-Multi [33]	Mixed	- (.833)	- (.503)
Brady et al [32]	MFCC	- (.846)	- (.450)
Weber et al. [31]	eGeMAPS	- (.793)	- (.456)
Somandepalli et al. [34]	Mixed	- (.800)	- (.448)
Han et al. [28]	13 LLDs	.666 (.755)	.364 (.476)
Proposed	raw signal	.715 (.786)	.369 (.428)

Table 5. Table V : RECOLA dataset results (in terms of ρ c subscript 𝜌 𝑐 \rho_{c} ) for prediction of arousal and valence. In parenthesis are the performance obtained on the development set. A dash is inserted if the results were not reported in the original papers.

Predictor	Features	Arousal	Valence
Baseline [29]	Geometric	.272 (.379)	.507 (.612)
RVM [30]	Geometric	- (.467)	- (.571)
Video CNN-L4 [33]	Mixed	- (.595)	- (.497)
Brady et al [32]	Appearance	- (.346)	- (.511)
Weber et al. [31]	Geometric	- (.476)	- (.683)
Somandepalli et al. [34]	Geometric	- (.297)	- (.612)
Han et al. [28]	Geometric	.265 (.292)	.394 (.592)
Proposed	raw signal	.435 (.371)	.620 (.637)

Table 6. Table VI : RECOLA dataset results (in terms of ρ c subscript 𝜌 𝑐 \rho_{c} ) for prediction of arousal and valence. In parenthesis are the performance obtained on the development set.

Predictor	Audio Features	Visual Features	Arousal	Valence
OA RVM-SR [30]	eGeMAPS ComParE	Geometric Appearance	.770 (.855)	.545 (.642)
Han et al. [28]	13 LLDs	Geometric	.610 (.728)	.463 (.544)
Proposed	raw signal	raw signal	.714 (.731)	.612 (.502)

Equations

$$(f\star h)(i,j)=\sum _{k=-T}^T\sum _{m=-T}^{T}f(k,m)\cdot h(i-k,j-m)$$

$${\mathbf{y}}_k=\mathcal{F}({\mathbf{x}}_k,\{W_k\})+\mathbf{h}(x_k)$$

$${\mathcal{L}}_c$$

$$=1-2{\sigma }_{xy}^2{\psi }^{-1}$$

$$\frac{\partial {\mathcal{L}}_c}{\partial \mathbf{x}}\propto 2\frac{{\sigma }_{xy}^2(\mathbf{x}-{\mu }_y)}{{\psi }^2}+\frac{{\mu }_y-\mathbf{y}}{\psi },$$

$${\mathcal{L}}_c=\frac{{\mathcal{L}}_c^a+{\mathcal{L}}_c^v}{2}$$

Full text

I Introduction

Emotion recognition is an essential component towards complete interaction between human and machine, as affective information is fundamental to human communication. Applications of emotion recognition can be found in different domains. For instance, emotion states can be used to monitor and predict fatigue state [1]. In speech recognition, emotion recognition can be used in call centres, where the goal is to detect the emotional state of the caller and provide feedback for the quality of the service [2].

The task of recognising emotions is challenging because human emotions lack of temporal boundaries and different individuals express emotions in different ways [3]. Although current work around emotion recognition was concentrated around infering the emotion of a subject out of its speech other modalities such as visual information (facial gestures) have also been used. With the advent of deep neural networks in the last decade a number of groundbreaking improvements have been observed in several established pattern recognition areas such as object, speech and speaker recognition, as well as in combined problem solving approaches, e.g. in audio-visual recognition, and in the rather recent field of paralinguistics.

Numerous studies have shown the favourable property of these network variants to model inherent structure contained in the speech signal [4], with more recent research attempting end-to-end optimisation utilising as little human a-priori knowledge as possible [5]. Nevertheless, the majority of these works make use of commonly hand-engineered features have been used as input features, such as Mel-Frequency Cepstral Coefficients (MFCC), Perceptual Linear Prediction (PLP) coefficients, and supra-segmental features such as those used in the series of ComParE [6] and AVEC challenges [7], which build upon knowledge gained in decades of auditory research and have shown to be robust for many speech domains.

Recently, however, a trend in the machine learning community has emerged towards deriving a representation of the input signal directly from raw, unprocessed data. The motivation behind this idea is that, ultimately, the network learns an intermediate representation of the raw input signal automatically that better suits the task at hand and hence leads to improved performance.

In this paper, we study automatic affect sensing using both speech and visual information in an end-to-end manner. Features are extracted from the speech signal using a CNN architecture designed for the audio channel and from the visual information using a ResNet-50 network architecture [8]. The output of these networks are fused together and fed to an LSTM to find the affective state of individuals. Contrary to the current practises, where each network is trained individually and the results are simply fed to a classifier our system is trained in an end-to-end manner. To our knowledge this is the first work in literature that applies such an end-to-end model for audio visual emotion recognition task. Furthermore, we suggest using explicit maximisation of the concordance correlation coefficient ($\rho_{c}$) [9] in our model and show that this improves performance in terms of emotion prediction compared to optimising the mean square error objective, which is traditionally used. Finally, by further studying the activations of different cells in the recurrent layers, we find the existence of interpretable cells, which are highly correlated with several prosodic and acoustic features that were always assumed to convey affective information in speech, such as the loudness and the fundamental frequency. A preliminary version of this work was presented in [10], where only the raw speech waveform was used. We extend this work by considering also the visual modality in an end-to-end manner.

To show the benefit of our proposed multimodal model we evaluated it in the REmote COLlaborative and Affective (RECOLA) database. A part of this database was used for the Audio/Visual Emotion Challenge and Workshop (AVEC) 2016. Our model is trained using the whole database. Results show that the multimodal model benefits from the two modalities by producing equal results for arousal and valence as the speech and visual networks, respectively. We compare the unimodal and the multimodal models using results obtained in the AVEC 2016 challenge. Only the papers that used the audio, visual or audiovisual modalities are considered. In order to perform a fair comparison we apply the proposed method on the test set of the AVEC challenge. As shown by our experiments our unimodal models produce the best results for both the speech and visual modalities.

The remainder of the paper is organized as follows. Section II reports related studies on emotion recognition using multiple modalities with DNNs. Section III introduces the multimodal model architecture. Section IV describes the dataset used for the experiments. Section V presents the experiments performed and reports the results. Finally, section VI concludes this paper.

II Related Work

The performance of pattern recognition models have been greatly improved with DNNs. Recently, a series of new neural network architectures have been revitalised, such as autoencoder networks [11], Convolutional Neural Networks (CNNs) [12], Deep Belief Networks (DBNs) [13] or memory enhanced neural network models such as Long Short-Term Memory (LSTM) [14] models.

These models have been used in various ways for multimodal recognition tasks such as in speech recognition. For instance, Ngiam et al. [15] proposed a Multimodal Deep Autoencoder (MDAE) network to extract features from audio and video modalities. First, a bimodal DBN was trained to initialize the deep autoencoder and then the MDAE was fine-tuned to minimize the reconstruction error of both modalities. In another study, Hu et al. [16] proposed a temporal multimodal network named Recurrent Temporal Multimodal Restricted Boltzmann Machine (RTMRBM) to model audiovisual sequence of data. Another task that DNNs have also been used is gesture recognition. In [17] the authors use skeletal information and RGB-D images to recognize gestures. More particularly, they use DBNs to process skeleton features and a 3D CNN for the RGB-D data. Temporal information is considered by stacking a Hidden Markov Model (HMM) on top.

The emotion recognition domain has highly benefited with the advent of DNNs. Some works explore deep learning approaches for speech emotion recognition. For instance, Han et al. [18] uses hand-crafted features to feed a DNN that produces a probability distribution over categorical emotion states. From these probabilities they compute statistics from the whole utterance and finally, they perform classification by training an extreme learning machine. Lim et al [19] after transforming the data using short time fourier transform, they used CNNs to extract high-level features. In order to capture the temporal structure LSTMs were used. In a similar work, Trigeorgis et al.[10] proposed an end-to-end model that uses a CNN to extract features from the raw signal and then an LSTM network to capture the contextual information in the data.

Other works try to solve the emotion recognition task by using facial information with DNNs. For example, Huang et al. [20] proposed a transductive learning framework for image-based emotion recognition by combining DNNs and hypergraphs. More particularly, after the DNN was trained for emotion classification task, each node in the last fully connected layer was considered as an attribute and used to form a hyperedge in a hypergraph. In another study, Ebrahimi et al. [21] combined CNNs and RNNs to recognise categorical emotions in videos. A CNN was first trained to classify static images containing emotion. Then, the extracted features from the CNN were used to train an RNN to produce an emotion for the whole video.

Recently, combining audio and visual modalities have great success for recognizing emotions. Some studies exploited the beneficial features DNNs can extract [22, 23, 24]. Kim et al. [23] proposed four different DBN architectures with one of them being a basic 2-layer DBN, and the others variation of it. The basic architecture first learns the features of the audio and video separately. After which it concatenates these features from the two modalities it uses them to learn the second layer. The features were evaluated using a Support Vector Machine (SVM). In another study, Kahou et al. [24] proposed to combine modality-specific DNNs to recognize categorical emotions in video. A CNN was used to analyze the video frames, a DBN to capture audio information, a deep autoencoder to model human actions depicted within the entire scene, and finally a CNN network to extract features from the mouth of the human. To output a final prediction they used two techniques that gave similar results. The first is to take the average of modality-specific predictions and in the second they learned an SVM with an RBF kernel using the concatenation features. In another study [25] compared hand-crafted features extracted from faces using multi-scale Dense SIFT features (MSDF), and features extracted from CNNs to train linear Support Vector Regression (SVR). The extracted audio features were the extended Geneva Minimalistic Acoustic Parameter Set (eGeMAPS). The combination of the features were used to learn a Support Vector Regression (SVR).

Zhang et al. [26] used a multimodal CNN for classifying emotions with audio and visual modalities. The model is trained in two phases. In the first phase the two CNNs are pretrained on large image datasets and fine-tuned to perform emotion recognition. The audio CNN takes as input the mel-spectogram segment of the audio signal and the video CNN takes the face. In the second phase a DNN was trained that comprised of a number of fully-connected layers. The concatenation of the features extracted by the two CNNs were inputted. In another study, Ringeval et al. [27] uses an BLSTM-RNN to capture the contextual information that exists in the multimodal features (audio, video, physiological) extracted from the data. In a more recent work, Han et al. [28] proposes the strength modeling framework which can be implemented as feature-level and decision-level fusion strategy and comprises of two regression models. The first model’s predictions are concatenated with the original feature vector and fed to the second regression model for the final prediction.

The importance of recognizing emotions motivated the creation of the Audio/Visual Emotion Challenge and Workshop (AVEC) [29]. In 2016 challenge audio, video and physiological modalities were considered. In one of the submitted models for the challenge, Huang et al. [30] proposed to use variants of Relevance Vector Machine (RVM) for modeling audio, video and audiovisual data. In another work model by Weber et al. [31] used high-level geometry features for predicting dimensional features. Brady et al. [32] also used low- and high-level features for modeling emotions. In a different study Povolny et al. [33] complemented original baseline features for both audio and video to perform emotion recognition. Somandepalli et al. [34] also used additional features but only for the audio modality.

All of the works in the literature make use of commonly hand-crafted features in audio or visual modality or in some cases in both of them. Moreover, they do not always consider temporal information in the data. In this study we propose a multimodal model trained end-to-end that also considers the contextual temporal information.

III Proposed Method

One of the first steps in a traditional machine learning algorithms is to extract features from the data. To extract features in audio, finite impulse response filters can be used which perform time-frequency decomposition to reduce the influence of background noise [35]. More complicated hand-engineered kernels, such as gammatone filters [36], which were formulated by studying the frequency responses of the receptive fields of auditory neurons of grassfrogs, can be used as well.

A key component of our model is the convolution operation. For the audio and visual signals, 1-d and 2-d convolution is used, respectively.

[TABLE]

where $f(x)$ is a kernel function whose parameters are learnt from the data of the task in hand. After the spatial-modeling of the signals, which removes background noise and enhances specific parts of the signals for the task in hand, we model the temporal structure of both speech and video by using a recurrent network with LSTM cells. We use LSTM for (i) simplicity, and (ii) to fairly compare against existing approaches which concentrated in the combination of hand-engineered features and LSTM networks. Finally, our model is subsequently trained with backpropagation using the objective function, cf. Equation 3.

III-A Visual Network

None of the first steps in the traditional face recognition pipeline is feature extraction utilizing hand-crafted representations such as Scale Invariant Feature Transform (SIFT) and Histogram of Oriented Gradients (HOG). Recently, deep convolutional networks have been used to extract features from faces [26].

In this study we use a deep residual network (ResNet) of 50 layers [8]. As input to the network we used the pixel intensities from the cropped faces of the subject’s video. Deep residual networks adopt residual learning by stacking building blocks of the form:

[TABLE]

where $\mathbf{x}$ and $\mathbf{y}$ are the input and output of the layer $k$, $\mathcal{F}(\mathbf{x}_{k},\{W_{k}\})$ is the residual function to be learned and $h(\mathbf{x}_{k})$ can be either an identity mapping or a linear projection to match the dimensions of function $\mathcal{F}$ and the input $\mathbf{x}$.

The first layer of ResNet-50 is a $7x7$ convolutional layer with 64 feature maps, followed by a max pooling layer of size $3x3$. The rest of the network comprises of 4 bottleneck architectures, where after these architectures a shorcut connection is added. These architectures contain 3 convolutional layers of sizes 1x1, 3x3, and 1x1, for each residual function. Table I shows the replication and the sizes of the feature maps for each bottleneck architecture. After the last bottleneck architecture an average pooling layer is inserted.

III-B Speech Network

In contrast to previous work done in the field of paralinguistics, where acoustic features are first extracted and then passed to a machine learning algorithm, we aim at learning the feature extraction and regression steps in one jointly trained model for predicting the emotion.

Input. We segment the raw waveform to 6 s long sequences after we preprocess the time-sequences to have zero mean and unit variance to account for variations in different levels of loudness between the speakers. At 16 kHz sampling rate, this corresponds to a 96000-dimensional input vector.

Temporal Convolution. We use $F=20$ space time finite impulse filters with a 5ms window in order to extract fine-scale spectral information from the high sampling rate signal.

Pooling across time. The impulse response of each filter is passed through a half-wave rectifier (analogous to the cochlear transduction step in the human ear) and then downsampled to 8 kHz by pooling each impulse response with a pool size = 2.

Temporal Convolution. We use $M=40$ space time finite impulse filters of 500ms window. These are used to extract more long-term characteristics of the speech and the roughness of the speech signal.

Max pooling across channels. We perform max-pooling across the channel domain with a pool size of 10. This reduces the dimensionality of the signal while preserving the necessary statistics of the convolved signal.

Dropout. Due to the large number of parameters (1.1 mil.) compared to the number of training examples we need to perform some regularisation in order for the model not to overfit on the training data. We opt to use dropout with a probability of 0.5.

III-C Objective function

To evaluate the agreement level between the predictions of the network and the gold-standard derived from the annotations, the concordance correlation coefficient ($\rho_{c}$) [9] has recently been proposed [37, 7]. Nonetheless, previous work minimized the MSE during the training of the networks, but evaluated the models with respect to $\rho_{c}$ [37, 7]. Instead, we propose to include the metric used to evaluate the performance in the objective function ($\mathcal{L}_{c}$) used to train the networks. Since the objective function is a cost function, we define $\mathcal{L}_{c}$ as follow:

[TABLE]

where $\psi=\sigma_{x}^{2}+\sigma_{y}^{2}+(\mu_{x}-\mu_{y})^{2}$ and $\mu_{x}=\operatorname{\mathbb{E}}(\mathbf{x})$, ${\mu_{y}=\operatorname{\mathbb{E}}(\mathbf{y})},{\sigma_{x}^{2}=\mbox{var}(\mathbf{x})},\sigma_{y}^{2}=\mbox{var}(y)$ and $\sigma^{2}_{xy}=\mbox{cov}(\mathbf{x},\mathbf{y})$ Thus, to minimise $\mathcal{L}_{c}$ (or maximise $\rho_{c}$), we backpropagate the gradient of the last layer weights with respect to $\mathcal{L}_{c}$,

[TABLE]

where all vector operations are done element-wise.

III-D Network Training

Before training the multimodal network, each modality-specific network is trained separately to speed up the training procedure.

Visual Network. For the visual network, we chose to fine-tune the pretrained ResNet-50 on the database used in this work. This model was trained on the ImageNet 2012 [38] classification dataset that consists of 1000 classes. The pretrained model was preferred than training the network from scratch in order to be benefited by the features already learned by the model. To train the network a 2-layer LSTM, with 256 cells each is stack on top of it to capture temporal information.

Speech Network. The CNN network operates on raw signal to extract features from. In order to consider the temporal structure of speech, we use two LSTM layers with 256 cells each on top of the CNN.

Multimodal Network. After training the visual and speech networks the LSTM layers are discarded and only the extracted features are considered. The speech network extracts 1280 features while the visual netowork 640 features. These are concatenated to form a 1920 dimensional feature vector and used to feed a 2-layer LSTM with 256 cells each. The LSTM layers are trained and the visual and speech networks are fine-tuned. Figure 1 shows the multimodal network.

The goal for each unimodal and the multimodal network is to minimize:

[TABLE]

where $\mathcal{L}_{c}^{a}$ and $\mathcal{L}_{c}^{v}$ are the concordance of the arousal and valence, respectively.

For the recurrent layers of speech, visual and multimodal networks, we segment the 6 s sequences to 150 smaller sub-sequences to match the granularity of the annotation frequency of 40 ms.

IV Dataset

Time-continuous prediction of spontaneous and natural emotions (arousal and valence) is investigated on speech and visual data by using the REmote COLlaborative and Affective (RECOLA) database introduced by Ringeval et al.  [39]; the full dataset for which participants gave their consent to share their data is used for the purpose of this study. Four modalities are included in the corpus: audio, video, electro-cardiogram (ECG) and electro-dermal activity (EDA). In total, 9.5h of multimodal recordings from 46 French-speaking participants were recorded and annotated for 5 minutes each, performing a collaboration task in dyads during a video conference. Among the participants, 17 were French, three German and three Italian. The dataset is split in three partitions – train (16 subjects), validation (15 subjects) and test (15 subjects) – by stratifying (i.e., balancing) the gender and the age of the speakers. Finally, 6 French -speaking annotators (three male, three female) annotated all the recordings.

V Experiments & Results

For training the models we utilised the Adam optimisation method [40], and a fixed learning rate of $10^{-4}$ throughout all experiments. For the audio model we used a mini-batch of 25 samples. Also, for regularisation of the network, we used dropout [41] with $p=0.5$ for all layers except the recurrent ones. This step is important as our models have a large amount of parameters ($\approx{1.5M}$) and not regularising the network makes it prone on overfitting on the training data.

For the video model, the image size used was $96\times 96$ with mini-batch of size 2. Small mini-batch is selected because of hardware limitations. The data were augmented by resizing the image to size $110\times 110$ and randomly cropping it to equal its original size. This produces a scale invariant model. In addition, color augmentation is used by introducing random brightness and saturation to the image.

Finally, for all investigated methods, a chain of post-processing is applied to the predictions obtained on the development set: (i) median filtering (with size of window ranging from 0.4 s to 20 s) [7], (ii) centring (by computing the bias between gold-standard and prediction) [42], (iii) scaling (using the ratio of standard-deviation of gold-standard and prediction as scaling factor) [42] and (iv) time-shifting (by shifting the prediction forward in time with values ranging from 0.04 s to 10 s), to compensate for delays in the ratings [43]. Any of these post-processing steps is kept when an improvement is observed on the $\rho_{c}$ of the validation set, and applied then with the same configuration on the test partition.

V-A Ablation study

Due to memory and training instability concerns [44] its not always optimal to use very large sequences in recurrent networks. The justification for this can be either the over-blowing of gradients or the very deep unrolled graph which makes training of such big networks harder.

In order to choose the best sequence length to feed our LSTM layers, we conducted experiments using sequence lengths 75, 150, and 300 for both speech and visual models. Table II shows the results on the development set. For all the experiments a total of 60 epochs were run for our models.

For the visual network we expect to get the highest value in the valence dimension, while for the speech model in the arousal dimension. Results indicate that the best value for the speech model is 150 while for the visual model is 300. Due to the fact that the difference in performance for the visual network is small when sequence length of 150 or 300 is used, we chose to train the mutlimodal network with 150 sequence length.

V-B Speech Modality

Results obtained for each method, using all 46 participants, are shown in Table III. In all of the experiments, our model outperforms the designed features in terms of $\rho_{c}$. One may note, however, that the eGEMAPS feature set provides close performance on valence, which is much more difficult to predict from speech compared to arousal. Furthermore, we show that by incorporating $\rho_{c}$ directly in the optimisation function of all networks allows us to optimise the models on the metric $(\rho_{c})$ on which we evaluate the models. This provides us with i) a more elegant way to optimise models, and ii) gives consistently better results across all test-runs as seen in Table III.

In addition, we compare the performance on the results obtained for methods that exist in the literature. Most of them have been submitted to the AVEC 2016 challenge, using 27 participants Table IV. In case performance on the test or validation set was not reported in the paper a dash is inserted. Results show that our model outperforms the other models in the test set when predicting arousal dimension. It is important to notice that although our model gets lower $\rho_{c}$ on the arousal dimension for the validation set compared to the baseline of the challenge, its performance is better on the test set.

V-B1 Relation to existing acoustic and prosodic features

The speech signals convey information about the affective state either explicitly, i.e., by linguistic means, or implicitly, i.e., by acoustic or prosodic cues. It is well accepted amongst the research community that certain acoustic and prosodic features play an important role in recognising the affective state [45]. Some of these features, such as the mean of the fundamental frequency (F0), mean speech intensity, loudness, as well as pitch range [46], should thus be captured by our model.

To gain a better understanding of what our speech model learns, and how this relates to existing literature, we study the statistics of gate activations in the network applied on an unseen speech recording; a visualisation of the hidden-to-output connections of different cells in the recurrent layers of the network is given in Figure 2. This plot shows that certain cells of the model are very sensitive to different features conveyed in the original speech wave form.

V-C Visual Modality

Visual modality has been shown to more easily predict the valence dimension rather than the arousal.  Table V presents the best results on the RECOLA dataset for the valence dimension. Only the work from Han et al. [28] was not submitted to the AVEC 2016 challenge. The features used for all of the models are appearance and geometric. For appearance Local Gabor Binary Patterns from Three Orthogonal Planes (LGBP-TOP) features were extracted, whereas facial landmarks were extracted for the geometric features. The input to our network are the raw pixel intensities from the face extracted from the frames of the videos using the Multi-Domain Convolutional Neural Network Tracker (MDNet) [47] tracking algorithm. This algorithm takes the bounding box of the face in the first frame of the video and tracks it in all frames.

As expected visual modality benefits the models in the valence dimension. The only exception is the Video CNN-L4 model which performs better in the arousal dimension when appearance features are used. Our model outperforms all the other models in the valence dimension for the test set.

V-D Multimodal Analysis

Only two other models found in the literature to use both speech and visual modalities on the RECOLA database. These are the Output-Associative Relevance Vector Machine Staircase Regression (OA RVM-SR) [30] and the strength modeling system proposed by Han et al. [28]. Results are shown in Table VI. Our model outperforms the other two models in the valence dimension with high magnitude. For the arousal dimension the OA RVM-SR produces the best results. However, we would like to note here that there are two main differences between our system and the system in  [30] (a) the system in  [30] was trained using both training and validation set, whereas our model was trained using only the training set and (b) our system operates directly on the raw pixel domain, while the system in  [30] made use of a number of geometric features (e.g., 2D/3D facial landmarks etc.) which require the presence of an accurate facial landmark tracking methodology (ours was applied on the results of a conventional face detector only). We expect that our results would be improved further by applying similar strategies. Finally, to further demonstrate the benefits of our model for automatic prediction of arousal and valence Figure 3 illustrates results for single test subject from RECOLA.

VI Conclusion

In this paper, we propose a multimodal system that operates on the raw signal, to perform an end-to-end spontaneous emotion prediction task from speech and visual data. To consider the contextual information in the data LSTM network was used. To speed up the training of the model we pretrained the speech and visual networks, separately. In addition, we study the gate activations of the recurrent layers in the speech modality and find cells that are highly correlated with prosodic features that were always assumed to cause arousal. Our experiments on the unimodal modality show that our models achieve significantly better performance in the test set in comparison to other models using the RECOLA database including those submitted to the AVEC2016 challenge, thus demonstrating the efficacy of learning features that better suit the task-at-hand. On the other hand, our multimodal model performs better in the valence dimension rather than the arousal compared to other models.

Acknowledgment

The support of the EPSRC Centre for Doctoral Training in High Performance Embedded and Distributed Systems (HiPEDS, Grant Reference EP/L016796/1) is gratefully acknowledged.