A Bayesian Deep Learning Framework for End-To-End Prediction of Emotion from Heartbeat

TL;DR

This paper introduces a Bayesian deep learning model that predicts emotional valence from heartbeat data, providing uncertainty estimates to improve real-world affective computing applications like healthcare.

Contribution

It presents an end-to-end neural network for emotion classification from heartbeat signals along with a Bayesian approach to quantify prediction uncertainty.

Findings

Achieved 90% classification accuracy on benchmark datasets.

Demonstrated effective uncertainty modeling for emotion predictions.

Showcased potential for non-invasive emotion monitoring in healthcare.

Abstract

Automatic prediction of emotion promises to revolutionise human-computer interaction. Recent trends involve fusion of multiple data modalities - audio, visual, and physiological - to classify emotional state. However, in practice, collection of physiological data `in the wild' is currently limited to heartbeat time series of the kind generated by affordable wearable heart monitors. Furthermore, real-world applications of emotion prediction often require some measure of uncertainty over model output, in order to inform downstream decision-making. We present here an end-to-end deep learning model for classifying emotional valence from unimodal heartbeat time series. We further propose a Bayesian framework for modelling uncertainty over these valence predictions, and describe a probabilistic procedure for choosing to accept or reject model output according to the intended application. We benchmarked our framework against two established datasets and achieved peak classification accuracy of 90%. These results lay the foundation for applications of affective computing in real-world domains such as healthcare, where a high premium is placed on non-invasive collection of data, and predictive certainty.

Tables

Table 1. TABLE I: Summary of relevant work - classification of emotion from heartbeat

Author	Stimulus	Subjects	LkSO	Model	Target	Performance
Katsigiannis & Ramzan 2018	Videos	23	No	SVM	High/Low Valence	F1. 0.5305 (Chance: 0.500)
Subramanian et al 2018	Videos	58	No	NB	High/Low Valence	Acc. 60% (Chance: 50%)
Miranda-Correa et al 2017	Videos	40	Yes	NB	High/Low Valence	F1. 0.545 (Chance: 0.500)
Guo et al 2016	Videos	25	No	SVM	High/Low Valence	Acc. 71.40 (Chance: 50%)
Ferdinando et al 2016	Videos & IAPS	27	Yes	KNN	High/Medium/High Valence	Acc. 59.2% (Chance: 33.3%)
Valenza et al 2014	IAPS	30	No	SVM	High/Low Valence	Acc. 79.15% (Chance: 50%)
Agrafioti et al 2012	IAPS	32	Yes	LDA	Gore, Erotica	Acc. 46.56% (Chance: 50%)

Table 2. TABLE II: Comparison of mean accuracy and F1 scores.

		[38]	[41]	LSTM (Non-Bayes)	CNN (Non-Bayes)	LSTM+CNN (Non-Bayes)	LSTM+CNN (Bayes $\alpha =$ 0.5)	LSTM+CNN (Bayes $\alpha =$ 0.9)
AMIGOS	Acc.	0.54	-	0.47	0.69	0.79	0.81	0.90
	F1	-	-	0.49	0.65	0.78	0.80	0.88
DREAMER	Acc.	-	0.62	0.61	0.61	0.70	0.71	0.86
	F1	-	0.53	0.52	0.54	0.66	0.66	0.83

Equations

$$\begin{array}{r}p(\mathbf{\theta }|\mathbf{D})=\frac{p(\mathbf{D}|\mathbf{\theta })p(\mathbf{\theta })}{p(\mathbf{D})}\end{array}$$

$$\begin{array}{r}p(y|\mathbf{x},\mathbf{D})=\int p(y|\mathbf{x},\mathbf{\theta })p(\mathbf{\theta }|\mathbf{D})d\mathbf{\theta }\end{array}$$

$$\begin{array}{r}p(\mathbf{D})=\int p(\mathbf{D}|\mathbf{\theta })p(\mathbf{\theta })d\mathbf{\theta }\end{array}$$

Full text

©2020 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media. DOI: 10.1109/TAFFC.2020.2981610

I Introduction

Humans are social creatures that evolved to think and communicate using emotional information. Cognition and emotion are thus intrinsically linked. Indeed, emotion has been shown to impact attention [1, 2, 3], memory [4, 5, 6, 7], perception [8, 9], and decision-making [10, 11, 12]. Automated analysis of human emotion has correspondingly garnered significant interest across academia and industry in recent years.

A wealth of research within the field of affective computing has focussed on the analysis of face, voice and text data [13, 14, 15, 16, 17, 18, 19, 20]. However, comparatively few studies investigate the prediction of emotion from physiological signals. The link between emotion and physiology has neurobiological foundations in the limbic and autonomic nervous systems. The limbic system, which includes structures such as the amygdala and hippocampus, is important for the processing of emotional information [21, 22, 23]. Physiological responses to emotional stimuli are then coordinated by another limbic structure, the hypothalamus, which regulates heartbeat through antagonistic activity in the sympathetic and parasympathetic branches of the autonomic nervous system (ANS) [24, 25].

For ease of reference, we refer to audio-, visual-, and physiology-based methods of emotion detection as EDA, EDV, and EDP respectively. EDP has tremendous potential to compliment alternative methods of affective computation. EDA and EDV rely heavily on expression, which varies across individuals and cultures [26, 27] and leaves room for deception. By comparison, physiological processes are far less volitional. EDP further presents an opportunity for non-invasive continuous monitoring. Physiological signals may be passively analysed throughout the day, whereas audiovisual data is rarely available or feasible to collect continuously. EDP therefore has the potential to fill critical gaps in domains such as healthcare (where physiological data is already abundant), transport, and hospitality.

To date, there exist a range of affordable wearable monitoring devices that possess the capacity for high quality heartbeat recording [28, 29]. These devices have already been used to detect cardiac abnormalities such as atrial fibrillation [30]. However, while heartbeat data is common, other physiological signals are strikingly rare in today’s technology landscape. To be immediately relevant, therefore, EDP systems must be able to generate accurate predictions using only unimodal heartbeat input in the form of inter-beat-intervals (IBIs). Furthermore, the cardiac cycle is a complex dynamical process. The corresponding heartbeat time series is non-stationary [31] and non-linear [32]. In order to adequately describe these characteristics, EDP systems must have the capacity to model complex temporal structure.

In many real-world scenarios, confidence is a key ingredient to decision-making (e.g. healthcare [33]). For applications in these domains, affective computing systems must have the capacity to describe uncertainty over their emotional state output. Consider a binary high/low valence classification problem. One might decide that valence predictions close to the class boundary (here, the valence scale mid-point) are more likely to be classified incorrectly. A solution could therefore be to only accept predictions that lie beyond a pre-specified distance from the class boundary (let’s call this the ‘acceptance boundary’). The question remains $-$ how does one choose the acceptance boundary in a meaningful way? (Fig. 1A). Furthermore, how do we translate this acceptance boundary to new environments where the data may have completely different statistical properties? Moreover, it is a strong assumption that predictions near to the class boundary are more likely to be incorrect (Fig. 1B). Indeed, a better approach would be to collect some measure of certainty in the model output. The decision on whether or not to accept the model’s prediction can then be specified according to model confidence, rather than the output value. This matches the intuition that confidence is the key ingredient to decision-making, and permits simple translation of a given acceptance boundary between different datasets.

In this study, we develop an end-to-end deep learning model for classifying emotional valence from unimodal IBI data. We implement recurrent and convolutional architectures to model temporal structure in the IBI signal, and further propose a Bayesian framework for modelling uncertainty in the output. We go on to describe a procedure for adjusting the acceptance boundary according to varying demands on confidence. This framework will be critical for applications of affective computing in domains such as healthcare, where a high premium is placed on predictive certainty and interpretability. We believe this is the first such model of its kind, has implications across the field of affective computing, and accelerates near-term relevance of EDP in real-world settings.

II Related Work

This section provides an overview of relevant work, with a focus on (A) analysis of unimodal heart data and temporal models for EDP, and (B) Bayesian neural networks.

II-A Analysis of Unimodal Heart Data and Temporal Modelling

Peripheral physiological markers of autonomic nervous activity include galvanic skin response (GSR), electroencephalogram (EEG), electromyogram (EMG), respiration, skin temperature (ST), electrocardiogram (ECG) and photoplethysmogram (PPG) [34]. Note that a heartbeat time series can easily be extracted from both ECG and PPG in the form of inter-beat-intervals (IBIs).

Existing approaches for EDP typically pool a number of biosignals to form a multimodal input to some classifier algorithm [35, 36, 37]. However, this multimodal approach contradicts the vast amount of physiological data obtained by affordable wearable devices today (which is almost exclusively unimodal IBI data). Comparatively few studies narrow their scope in accordance with these practical limitations.

Those studies that have explored unimodal IBI models for emotion detection tend to ignore temporal structure of the signal. Instead, they employ ‘static’ classifiers that process global features from the input time series (or features calculated over a number of shorter segments). Such approaches include Naive Bayes (NB) [38, 39], linear discriminant analysis (LDA) [40], and support vector machine (SVM) [41, 42, 43]. A summary can be found in Table I.

A number of studies have sought to model temporal information within EEG signals, using hidden Markov models [44], Gaussian Process models [45], continuous conditional random fields [46], and long short-term memory (LSTM) neural networks [47]. Such temporal treatment, however, is rare for other peripheral physiological signals. One notable exception to this involved the use of a temporal neural network to predict valence from ECG input [48]. Here, a combination of convolutional and recurrent layers performed end-to-end learning, improving on computationally expensive manual feature engineering schemes. In this study, we too implement end-to-end learning. However, in accordance with aforementioned practical limitations, we further limit model input to IBI time series in order to simulate data of the form generated by consumer wearables.

Convention within affective computing is to implement an experimental protocol where an emotion-inducing stimulus is combined with participant self-reporting of affective state. However, there is a large amount of variability between self-reporting frameworks. A clear division concerns discrete and dimensional theories of emotion [49]. To construct our Bayesian framework, we require only a clear measure affective state that can be easily compared with existing benchmarks. For the purposes of this study, we choose to use emotional valence, which is suited to both a classification and regression setting, and avoids issues of semantic confusion that can accompany the terminology around emotional/physiological arousal in the cardiovascular literature.

At this point, we wish to point out a stark absence of consensus within the literature around data subsetting for machine learning. A typical experimental setup can yield multiple input-output pairs for a single study participant. Many studies partition train, validation, and test datasets without reference to the study participant from which the data was generated. However, ECG has been shown to exhibit subject-specificity [50]. It is therefore unsuitable to include data from a given participant in both the train and test/validation subsets. Moreover, real-world applications may not permit subject-specific calibration, making models that can generalise to new individuals increasingly useful. For these reasons, we propose a sensible evaluation method to be leave-k-subjects-out (LkSO) cross-validation, which has been used previously [38, 51, 40] and will be adopted in this study.

II-B Bayesian Neural Networks

Despite the widespread success of deep learning, traditional neural networks lack probabilistic considerations. This results in the model outputting a single point estimate for each input sample, which can lead to overconfidence in downstream decision-making. This is a serious issue for applications where datasets are imbalanced and representing uncertainty is of critical importance (e.g. medical diagnosis) [52].

To combat this, neural networks may be re-cast as Bayesian models in order to capture probability in the output. In this formalism, the weights of the network, $\boldsymbol{\theta}$, are said to belong to some prior distribution, $p(\boldsymbol{\theta})$. A posterior distribution over the network weights can then be conditioned on the data, $\boldsymbol{D}$, according to Bayes’ rule:

[TABLE]

where $p(\boldsymbol{\theta}|\boldsymbol{D})$ is the posterior distribution of the network weights and $\boldsymbol{D}$ are the training data, which for a supervised task comprises the input-output sample pairs (in our case, the input is the IBI time series and the output is the subject self-reported valence).

When making a prediction on a new input sample, $\mathbf{x}$, we can measure our uncertainty in the output, $y$, using the posterior predictive distribution, $p(y|\mathbf{x},\boldsymbol{D})$, which is calculated by marginalising out the network parameters:

[TABLE]

While useful from a theoretical perspective, such Bayesian treatment of neural networks is infeasible to compute. Indeed, the evidence term in the denominator of Equation 1 (also known as the marginal probability) amounts to the integral over all possible values of the network weights:

[TABLE]

For obvious reasons, exact posterior inference is rarely achievable. Instead, we seek to approximate these posterior distributions. Early attempts at this include Monte Carlo (MC) [53] or Laplace [54] approximation methods. However, these are slow and computationally expensive when applied to modern deep learning architectures. Research in the field has focussed on identifying faster inference methods such as stochastic gradient Langevin diffusion [55], expectation propagation [56], and variational methods [57].

Interestingly, Bayesian neural networks can also be constructed using Monte Carlo dropout - a widely-used approach to reduce over-fitting during network training [58]. Dropout is a process by which individual nodes within the network are randomly removed during training according to a specified probability [59]. By implementing dropout at test (as well as in training) and performing $N$ stochastic forward passes through the network, we can approximate a posterior distribution over model predictions (approaching the true distribution as $N\to\infty$ ). In this paper, we implement Monte Carlo dropout as an efficient way to describe uncertainty over emotional state predictions.

III Model Architecture

An overview of our model architecture is shown in Fig. 2. Data flows through two concurrent streams. One stream comprises four stacked convolutional layers that extract local patterns along the length of the time series. Each convolutional layer is followed by dropout and a ReLU activation function. A global average pooling layer is then applied to reduce the number of parameters in the model and decrease over-fitting. The second stream comprises a bidirectional LSTM followed by dropout. This models both past and future sequence structure in the input. The output of both streams is then concatenated before passing through a dense layer to output a regression estimate for valence.

IV Bayesian Framework

In order to capture uncertainty in model predictions, dropout is applied at test time. For a single input sample, stochastic forward propagation is run $N$ times to generate a distribution over model output. This empirical distribution approximates the posterior probability over valence, given the input IBI time series.

For regression problems, the reader may stop here. In order to translate from a regression to a classification scheme (as was used in [38] and [41], and will be implemented in this study for comparison), we introduce class boundaries in continuous space. For a binary classification scheme, the class boundary is naturally along the central point of the valence scale to delimit two class zones (high and low valence). We next introduce a confidence threshold, $\alpha$, which acts as the acceptance boundary and determines whether a prediction is accepted or rejected according to how confident we wish to be. For example, when $\alpha=0.95$, we specify that at least $95\%$ of the output distribution mass must lie within a given class zone in order for the input sample to be classified as belonging to that class (Fig. 1C). If this is not the case, no prediction is made (i.e. the model answers, ‘I don’t know’). As our model may not classify all instances, we also adopt the term ‘coverage’ to denote the proportion of samples for which it is confident enough to make a prediction.

Note that for a binary classification problem, and where $N$ is an odd number, there will always be at least 50% of the output distribution mass within one of the two class zones. Thus, when $\alpha=0.5$, classification is determined by the median of the output distribution (Fig. 1), and the coverage is 100%. As $\alpha$ increases, model behaviour moves from risky to cautious $-$ lower coverage, but predictions are more certain.

V Data

We applied our Bayesian deep learning framework for end-to-end prediction of emotion using heartbeat (IBI) data extracted from two established datasets $-$ AMIGOS [38] and DREAMER [41]. In this section, we provide details on these data, which were chosen for their quality, clarity, and comparability.

V-A AMIGOS

These data include 40 healthy participants (13 female; 27 male) aged between 21 and 40 years old (mean: 28.3). ECG data was recorded using a ShimmerTM ECG wireless monitoring device (256 Hz, 12 bit resolution). Subjects watched 16 short videos (duration $<250$ seconds) that had been previously scored for emotional content. The videos were presented in a random order with each trial comprising a 5-second baseline recording showing a fixation cross, presentation of the video stimulus, followed by self-assessment of valence on a scale of 1 to 9 using the self-assessment manikin (SAM) [60].

V-B DREAMER

These data include 25 healthy participants (11 female; 14 male) aged between 22 and 33 years old (mean: 26.6). ECG data was also recorded using a ShimmerTM ECG wireless monitoring device (256 Hz, 12 bit resolution). Subjects watched 18 short film clips (duration: $<395$ seconds), which had been previously scored for their ability to elicit emotional responses [61]. Each film clip was followed by self-assessment of valence on a scale of 1 to 5 using SAM [60], and was preceded by a neutral video presentation to establish baseline emotional state [61].

VI Methods

VI-A Pre-processing

To obtain data of the form generated by consumer wearables, IBIs were extracted from the ECG time-series using a combined adaptive threshold approach [62]. This markedly reduced the information content of the input signal, making our task more challenging than described in [48]. Nevertheless, inter-beat dynamics have previously been shown suitable for emotional state classification [43]. The IBI time series were z-score normalised and zero padded to the length of the longest training sample.

VI-B Training and Hyperparameters

Parameter search was used to select model hyperparameters. For this, a LkSO validation set of 4 subjects (k=4) was used to assess best-performance (lowest mean-squared loss) for a given combination of hyperparameters. Convolution kernels were initialised as He normal [63] with filter size set to 128, and window size decreasing from 8 to 2 time steps with network depth. 50% dropout was applied after each convolutional block, and 80% dropout followed the bi-directional LSTM comprising 32 hidden units. Training was run for 1500 epochs using Adam optimisation [64]. Learning rate decreased from $e^{-3}$ to $e^{-4}$, halving with a patience of 100 epochs. Final model parameters were set to those associated with the lowest mean-squared loss on the validation set during training. The model was implemented using Tensorflow [65].

VI-C Evaluation

As discussed in Section II-A, model performance was assessed using 10-fold leave-one-subject-out cross validation in order to generalise to new participants. Dropout was applied at test time with $N=1001$ forward propagations made through the network to generate an empirical distribution over model output. In accordance with the original studies from which we obtained our data, labels for valence were divided into high and low classes using the midpoint value of the SAM scale (5 for AMIGOS; 3 for DREAMER). As outlined in Section III, a given test input sample was classified as high/low valence provided a proportion of at least $\alpha$ posterior distribution mass fell within a given class zone. If this was not the case, no prediction was made. We demonstrate the effect of adjusting $\alpha$ in Section VII. Model accuracy was then calculated as the fraction of correct classifications over total predictions covered by the model.

VII Results

To identify the benefit conferred by our temporal network architecture, we first evaluated our model without dropout at test time. In this non-Bayesian setting, model output was a single point estimate, which fell either in the high or low class zones, and was classified accordingly. Here, we achieved higher accuracy across both datasets than previously reported [38, 41] (Table II). To further assess the effect of each network stream, we evaluated the CNN and LSTM stream separately (Table II). The combined LSTM+CNN consistently outperformed either stream alone, indicating that the two streams each extract unique information relevant to the classification task.

We next implemented our Bayesian framework with confidence threshold set to 50% ($\alpha=0.5$). As expected, maximal model coverage was observed (Fig. 3B). Furthermore, no significant difference was found between the non-Bayes model and the Bayesian model when $\alpha=0.5$ (Mann-Whitney U test; $p>0.1$), demonstrating that our Bayesian framework does not lead to any performance decrease when coverage is 100% (Table. II). For clarity, recall that $\alpha$ denotes our confidence threshold on the model output, and is not used to denote the significance level of hypothesis tests.

As the certainty threshold $\alpha$ increases, so too does classification accuracy, demonstrating a clear relationship between model confidence and propensity to make accurate predictions (Fig. 3A, and Table II). Naturally, as $\alpha$ increases, model coverage decreases due to the fact that fewer output distributions meet the necessary threshold for a prediction to be made. We see that with a 90% confidence threshold ($\alpha=0.9$), our model achieved peak accuracy for both datasets (Fig. 3A, and Table II). Interestingly, we found that certainty over model output was significantly greater for input time series that belong to the low valence class, for both datasets, as shown by Mann–Whitney U test (Fig. 3 G,H). This pattern is also reflected in the consistently better performance observed for the low valence class (Fig. 3 C,D,E,F).

We went on to plot valence predictions according to whether the prediction was accepted or rejected by our Bayesian framework. In Fig. 4, we see that the two groups are not linearly separable in the valence space, highlighting that our acceptance boundary $-$ which is based on model certainty $-$ is not equivalent to simply ignoring model predictions close to the class boundary. The decision to reject or accept is a fundamental property of the model confidence in its own output.

VIII Discussion

The growing prevalence of high-fidelity, affordable wearable monitoring devices creates an opportunity for continuous emotion detection ‘in the wild’. The vast majority of existing approaches in the literature have relied on the fusion of multiple physiological signals for physiology-based emotion detection (or EDP). Although this multimodal treatment provides significant performance benefits, it is limited in practice. Indeed, the vast amount of physiological data obtained by affordable wearable devices today is almost exclusively unimodal IBI time series. Timely application of EDP in real world settings therefore requires models that are compatible with these input signals.

It has been shown previously that IBI extracted from PPG corresponds closely with IBI extracted from ECG [66, 67]. This allowed us to exploit existing high-quality ECG datasets for the purposes of this study. We developed an end-to-end neural network to model temporal structure in the IBI time series and classify emotional valence. This outperformed previous benchmarks on these datasets [38, 41] and accelerates near-term relevance of EDP in real-world settings.

Using Monte Carlo dropout, we re-cast our model as a probabilistic neural network in order to approximate the posterior predictive distribution over the output. This provides a crucial measure of uncertainty in the model predictions, which we used to specify a statistically meaningful acceptance boundary. By tuning the confidence threshold, $\alpha$, our Bayesian framework is able to trade off performance against coverage ($\alpha$ is itself a hyper-parameter for future cross-validation). In this study, we report peak model accuracy of 90%. Further flexibility was achieved by framing our model as a regression problem, which allows the experimenter to specify class boundaries appropriate for binary- or multi-class tasks.

Incorporating Bayesian considerations will drastically improve the applicability of affective computing in environments where confidence is a key ingredient to decision-making. Of particular interest to these authors is the application of emotion prediction to produce novel metrics within mental healthcare. For example, diagnosing the onset of major depressive disorder based on persistently low valence would reasonably require high levels of certainty. Additionally, clinical triaging systems based on predicted affective states may send uncertain predictions to a human expert for review. Similar levels of certainty may not, however, be necessary in consumer products. For example, content recommendations based on user mood might tolerate lower accuracy to ensure a greater number of recommendations. Our Bayesian framework provides a method of adjusting the acceptance boundaries in these different applications in a statistically meaningful way.

Furthermore, generating a probability distribution over model outputs could provide a sensible way to combine spatiotemporal and multimodal data. Indeed, these signals could be analysed by separate neural networks, and the resulting predictions combined via Bayesian integration (which would necessarily require certainty measures of the kind generated by our Bayesian framework).

Finally, our probabilistic framework provides deeper insight into the underlying properties of the data analysed. We found our model was more certain when classifying the low valence class for both datasets. This could be attributed to the subjective certainty of participants during their own self-reports. Alternatively, signatures of low valence within the heartbeat signal might contain more information than their high valence counterparts, requiring larger volumes of training data to reduce the uncertainty in the posterior predictive distribution. We look forward to future application of our Bayesian framework, and end-to-end deep learning model, to studies of emotional arousal, discrete models of emotion, multimodal input and indeed multimodal output.

IX Conclusions

In this study, we developed an end-to-end deep learning model for classifying emotional valence from unimodal heartbeat data. Our temporal neural network architecture outperformed previous models on the AMIGOS [38] and DREAMER [41] datasets. We further proposed a Bayesian framework for modelling uncertainty over emotional state predictions, providing a means to tune prediction acceptance boundaries in a statistically meaningful way according to the confidence requirements for different tasks. That model performance improved with increasing certainty thresholds ($\alpha$) illustrates the potential for probabilistic modelling to enhance affective computing systems. Taken together, the component parts of this study represent an important step towards application of emotion detection in real-world settings, and provide a probabilistic standard for future work.