Annotating and Modeling Empathy in Spoken Conversations

TL;DR

This paper develops an annotation scheme and an automatic system to recognize empathy in spoken conversations, addressing the challenge of operationalizing and measuring empathy in behavioral analysis.

Contribution

It introduces a novel empathy annotation protocol based on the modal model of emotions and an automatic classification system utilizing lexical and acoustic features.

Findings

The annotation scheme was validated on real-life conversations.

The automatic system outperformed baseline methods in call center data.

Features from speech and language levels improve empathy recognition.

Abstract

Empathy, as defined in behavioral sciences, expresses the ability of human beings to recognize, understand and react to emotions, attitudes and beliefs of others. The lack of an operational definition of empathy makes it difficult to measure it. In this paper, we address two related problems in automatic affective behavior analysis: the design of the annotation protocol and the automatic recognition of empathy from spoken conversations. We propose and evaluate an annotation scheme for empathy inspired by the modal model of emotions. The annotation scheme was evaluated on a corpus of real-life, dyadic spoken conversations. In the context of behavioral analysis, we designed an automatic segmentation and classification system for empathy. Given the different speech and language levels of representation where empathy may be communicated, we investigated features derived from the lexical and acoustic spaces. The feature development process was designed to support both the fusion and automatic selection of relevant features from high dimensional space. The automatic classification system was evaluated on call center conversations where it showed significantly better performance than the baseline.

Tables

Table 1. Table 1: An excerpt from a telephone conversation where the agent ( A ) is empathic towards a customer ( C ). The agent perceives the customer’s feeling and proactively takes actions to cope with customer’s emotional discomfort. English translations are in italics.

Dialog excerpt	Notes
\pbox7cm C: Ascolti … io ho una fattura scaduta di 833 euro vorrei sapere … tempo in cui posso pagarla.
(Listen… I have an 833 euros overdue bill… I would like to know… the time left to pay it.)
A: Ma perché non ha chiesto il rateizzo di questa fattura? Proviamo a far il rateizzo, ok? Così gliela blocco e lei ha più tempo per effettuare il pagamento.
(Why did not you ask to pay it in installments? We try to divide it into installments, is it ok for you? So I stop the overdue notices and you will have more time to pay)	\pbox4cmThe tone of voice and the hesitations of the customer show that she is not angry, she is ashamed for not being able to pay immediately the bill. This causes an empathic reply in the Operator’s attitude.
The selection of the speech act (question instead of authoritative declarative), the rhetorical structure of the second question, the lexical choice of ‘‘proviamo’’, instead of - for instance, ‘‘adesso provo a vedere...’’, all these contribute to prevent the customer’s feeling of being inadequate or ashamed.

Table 2. Table 2: Co-occurrence matrix between Empathy or Neutral on the agent channel (columns) and Anger or Frustration on the customer channel (rows).

	Empathy (%)	Neutral(%)
Anger or Frustration	156 (8.24)	246 (12.99)
Neutral	369 (19.48)	1123 (59.29)
Total	525 (27.72)	1369 (72.28)

Table 3. Table 3: Statistical significance (two-tailed two-sample t-test) of acoustic features is reported for each category ( rows ). In the second and third column we report largest p-value and the range of effect size (Cohen’s d ) for female and male speakers, respectively. The value of d, 0.2 0.2 ~{}0.2 , 0.5 0.5 ~{}0.5 and 0.8 0.8 ~{}0.8 denotes small, medium and large effect sizes, respectively [ 71 ] . The t-value provides the t-statistic of the t-test. The greater, i.e., either positive or negative, the value of the t-statistic, the greater the evidence against the null hypothesis. The closer to zero the more likely that there is no significant difference.

Feature type	Female	Male
\pbox5cmPitch (F0, voice-probability, voice-quality)	p<=6.04E-06	p<=6.73E-03
	d=[0.3 to 1.1]	d=[0.2 to 1.5]
	t=[-3.5 to 14.0]	t=[-2.2 to 15.0]
Loudness	p=3.08E-25	p=1.86E-29
	d=0.7	d=1.1
	t=9.1	t=10.9
Zcr	p=4.81E-08	p=2.01E-05
	d=0.3	d=0.4
	t=-3.6	t=-3.3
\pbox5cmSpectral (Energy in bands: 0-650 Hz, 250-650Hz, 1-4kHz; Roll-off points 25%, 50%, 75%, 90%; Position of spectral maximum and minimum, Centroid, Flux)	p<=5.66E-03	p<=3.64E-04
	d=[0.2 to 1.4]	d=[0.2 to 1.8]
	t=[-17.6 to 17.6]	t=[-15.0 to 16.8]
\pbox4cmAuditory-spectrum bands 0-25 for male and 0-21 for female	p<=6.25E-04	p<=1.26E-04
	d=[0.2 to 0.7]	d=[0.3 to 1.0]
	t=[1.1 to 8.9]	d=[2.9 to 9.6]

Table 4. Table 4: Statistical significance (two-tailed two-sample t-test) of lexical features. p: p-value of t-test, d: effect size, t: t-stat.

Lexical Features	p	d	t
vediamo un po (let’s see)	1.6E-03	1.0	-3.3
vediamo un attimino (let’s see a bit)	2.7E-03	1.7	-3.6
vediamo subito allora (let’s see now, then)	9.7E-03	3.8	-4.6
vedere (see)	3.3E-05	1.1	-4.4
facciamo (let’s do)	6.1E-07	1.3	-5.5
controlliamo (let’s check)	1.2E-03	0.7	-3.3
non si preoccupi (do not worry)	5.0E-03	1.1	-3.0
allora vediamo (so, let’s see)	4.2E-04	0.9	-3.7
assolutamente (absolutely)	4.1E-06	1.8	-5.4

Table 5. Table 5: Segment duration statistics (seconds) of the Neutral and Empathy segment-pairs and the total amount of speech for each category (hours).

Class	Avg. (s)	Std. (s)	Total (h)	# of Seg
Empathy	19	13	3	526
Neutral	220	148	32	526

Table 6. Table 6: Extracted low-level acoustic features

Low-level acoustic features
Voice Quality
Probability of voicing, jitter-local, jitter-DDP, shimmer-local,
log harmonics-to-noise ratio (HNR)
Cepstral
MFCC 1-14
Spectral
Auditory spectrum (RASTA-style) bands 0-25 (0-8kHz),
Spectral energy 250-650Hz, 1-4kHz,
Spectral roll-off points (0.25, 0.50, 0.75, 0.90),
Spectral slope 0-500Hz and 500-1500Hz,
Spectral flux, centroid, entropy, variance, skewness, kurtosis,
Spectral slope,
Difference of spectral flux between two consecutive frames
Psychoacoustic spectral sharpness, harmonicity
Alpha Ratio - ratio of the total energy from 50-1000Hz, and 1-5 Hz
Relative energy of formant 1, 2 and 3
First and second harmonic difference
First and third harmonic difference
Hammarberg index
Prosody
F0 final, F0 envelope, F0final with non-zero frames,
Root-mean-square signal frame energy,
Sum of RASTA-style auditory spectra,
Loudness, Zero crossing rate,
Formant frequencies [1-4], bandwidths [1-4]

Table 7. Table 7: Statistical functionals

Low-level acoustic features
Voice Quality
Statistical functionals
Percentile 1%, 99% and percentile range 1%-99%
Quartile (1-3) and inter-quartile (1-2, 2-3, 3-1) ranges
Relative position of max, min, mean and range
Arithmatic mean, root quadratic mean
Mean of non-zero values (nnz)
Contour centroid, flatness Std. deviation, skewness, kurtosis
Uplevel time 25, 50, 75, 90,
Rise time, fall time, left curvature time, duration
Mean, max, min and Std. deviation of segment length
Linear prediction coefficients (lpc0-5), lpc-gain
Linear regression coefficients (1-2) and error
Quadratic regression coefficients (1-3) and error

Table 8. Table 8: Empathy classification results at the segment level using acoustic, lexical, and psycholinguistic (LIWC) features together with feature and decision level fusion. Maj - Majority voting. Dim. - Feature dimension.

Experiments	Dim.	Test-Set
Random baseline		49.3
Acoustic	2400	68.1
Lexical	8000	65.6
LIWC	89	67.3
Acoustic+Lexical	2600	68.3
Maj(Acoustic+Lexical+LIWC)		70.1

Equations

$$\begin{array}{r}Cohe{n}^{\prime }sd=\frac{m1-m2}{{\sigma }^p};{\sigma }^p=\sqrt{\frac{\sigma {}_1^2+{\sigma }_2^2}{2}}\end{array}$$

$$\begin{gathered} tf\times idf= \\ log(1+f_{ij})\times log\left(\frac{\text{number of segments}}{\text{number of segments that include word i}}\right) \end{gathered}$$

$$W[F]=P(x|nearestmiss)-P(x|nearesthit)$$

$$H(s)=c_{\hat{j}};where\hat{j}=argma{x}_j\sum _{i=1}^T{h}_i^j(s)$$

$$C(f)=\left\{c_{i,j}(f)=\sum _{s\in S_T}\left[((y=i)\wedge (f(s)=j))\times length(s)\right]\right\}$$

Full text

††journal: Journal of Computer Speech and Languagemyfootnotemyfootnotefootnotetext: © 2017. This manuscript version is made available under the CC-BY-NC-ND 4.0 license http://creativecommons.org/licenses/by-nc-nd/4.0/

1 Introduction

Research in human emotions is an interdisciplinary field of study with relevant contributions from different scientific fields including neuroscience, psychology, and more recently computer science. In the latter discipline, the motivations for studying human emotions are driven by the goal of training virtual agents and improving human-computer interaction. Training intelligent agents to understand human emotions poses fundamental research challenges in a) the annotation of emotion examples for computers to learn from, b) the design of algorithms to automatically recognize and c) synthesize emotions. For example, in human-machine dialog research there has been growing interest in how to coordinate linguistic and paralinguistic signals and their impact on the performance of dialogs [1]. Progress has been reported in modeling full spectrum of human short and long-term states such as mood and emotion, and in modeling personality traits [2] by analyzing paralinguistic phenomena [3], facial expressions, gestures [4] and bio-signals [5]. Despite this growing interest there are very few studies that explicitly investigate complex emotions such as empathy.

The case for empathy is compelling because the ability to recognize and reproduce empathic behaviour of conversants would greatly improve the applicability of virtual agents. The concept of empathy was investigated in experimental psychology starting at the beginning of last century, when Titchener in $1909$ [6] coined the very word empathy as a translation of the German term Einfühlung. Since then this concept has been widely used to refer to a range of pro-social emotional behaviors, from sympathy to compassion, including accurate understanding of the other person’s feelings. Recently, the hypothesis that one’s empathy is triggered by understanding and sharing others’ emotional states has found neuroscientific underpinnings in the discovery of the mirror-neurons system. This system is hypothesized to embody unconscious routines of emotional and empathic behaviors in interpersonal relations. These include action understanding, attribution of intentions (mind-reading), and recognition of emotions and sensations [7]. In everyday life, empathy supports important aspects of inter-personal communication to an extent where some psychic diseases that affect the relationship with other persons, such as autism and Asperger syndrome, are explained in terms of impairment of empathic ability [8]. In computer science literature, the authors in [9] evaluate the synthesis of empathic virtual agents, while  [10] recognize the empathy of a therapist during interactions with his patients, and  [11] estimate the empathic behavior in meeting conversations.

Our interest in the problem of empathy is driven by the goal of learning computational models of complex emotional states on a psychologically motivated model of human emotions. In this paper, we refer to the psychological definition of empathy by Hoffman, who defines it as “an emotional state triggered by another’s emotional state or situation, in which one feels what the other feels or would normally be expected to feel in his situation” ([12]). In particular, we focus on identifying the instances of empathy emerging in spoken real-life conversations. To develop a computational model of empathy, we designed an experimental paradigm to observe the unfolding of emotions in the context of empathic behavior. For that we adopt the modal model of emotions by Gross [13] as a promising framework for defining an operational concept of empathy. We use this framework for developing the annotation guidelines for the annotation of empathy in a real-life spoken conversation corpus. Our research is organized into three phases: data observation and analysis, corpus annotation, and automatic classification. The guidelines for the annotation of the spoken corpus recommend a continuous selection of the emotion perception on the speech channel, and the assignment of discrete labels.

The main contributions of this paper are 1) the design and evaluation of an annotation model of empathy in spoken dialogues, 2) the training of an automatic classification system for empathy based on acoustic, lexical and psycholinguistic features111Psycholinguistic features are extracted from transcriptions. It is a knowledge-based approach, in which a lexicon is used to compute the correlation between words and psycholinguistic categories. More details can be found in Section 6.5.3., and 3) the evaluation of the classification system on a corpus of human-human spoken conversations. To the best of our knowledge, this is the first published research on perceiving, annotating and automatically recognizing empathy from human-human spoken conversations in real-life interactions.

The paper is organized as follows. In Section 2, we provide a review of relevant research on empathy. We then briefly discuss the modal model of emotions and its relevance to our work in Section 3. In Section 4, we describe the proposed annotation scheme for the annotators. In Section 5, we provide the corpus analysis and investigate the speech correlates of empathy. In Section 6, we describe the computational architecture of the automatic segmentation and classification system. We then provide the results and discussion of our experiments in Section 6.8. We discuss the research challenges and future work in Section 6.9.

2 Background Research

2.1 Psychology and Neuroscience Research

Over the past decades there have been significant efforts in investigating empathy in the fields of psychology and neuroscience [14] [15]. The complexity of the neural and psychological phenomena to be accounted for is huge and, in part, that complexity explains the existence of several psychological definitions of empathy. For example, the work in [16] accounts for different empathic phenomena occurring in the literature on empathy, and [17] examines eight distinct phenomena commonly labeled as empathy including emotional contagion, sympathy, projection, and affective inferential processes. Decety and Lamm [18] observe that some of the different definitions of empathy may share the underlying concept of “[…] an emotional experience that is more congruent with another’s situation than with one’s own”. The authors also state that empathic emotional experiences imply self-other differentiation, as opposed to emotional contagion. Among the different definitions of psychological theory there are some common features. Most of the definitions describe empathy as a type of emotional experience and/or emotional state. Moreover, the different definitions can be divided into two main classes. One encompasses the cognitive aspects of empathic behavior, such as one’s ability to accurately understand the feelings of another person. The other class entails sharing or the subject’s internal mimic of such feelings such as sympathy, emotional contagion, and compassion. The work in [19] takes a different perspective on empathy by focusing on the role of motivations for explaining a particular feature of the empathic experience, i.e., its oscillation between automaticity and dependency from the situational context.

2.2 Computational Models

2.2.1 Research in Speech Communication

Computational models of emotional states are needed to design machines that can understand and interact affectively with humans. Different signal components have been considered for analyzing the emotional instances in speech. Both verbal and non-verbal levels of spoken communication [20] have been considered since both are suggested to embody the expressive potential of language. Major focus has been devoted to the paralinguistic features of emotional speech, on the basis of the experimental evidence that emotional information is mostly conveyed by those levels (see [21] for a state-of-the-art review).

In the field of spoken language processing there are several collections of emotional annotated corpora. The authors in [22] report a significant disparity among those corpora in terms of complexity of the annotated emotions, explicitness of the emotion definitions, and identification of the annotation units. Most emotional corpora have been designed to perform specific tasks such as emotion recognition or emotional speech synthesis [23, 24]. The associated annotation schemes often depend on the specific tasks as well. The HUMAINE project [25] and the emotion multilingual collection in [26] base their annotation schemes on sets of discrete emotional lexical items or basic emotions.

Providing explicative models for annotating the emotional process itself in naturally occurring conversations is still an open challenge. Efforts in this direction are currently being made in the affective computing research, where awareness about the need for continuous annotation is increasing. The models that foster this approach, such as those discussed in [27] and [28], require annotators to continuously assign values to emotional dimensions and sets of emotional descriptors. Metallinou and Narayanan [29] emphasize that continuous annotation has several benefits, as well as some open challenges. One interesting finding is that continuous annotation may show regions which are characterized by perceived transitions of the emotional state. In [29], authors report a high number of factors that may affect inter-annotator agreement, such as person-specific annotation delays and confidence in understanding emotional attributes. There are very few studies in terms of empathy classification and most of them are carried out within controlled scenarios. Kumano et al. [11] studied four-party meeting conversations to estimate and classify empathy, antipathy and unconcerned emotional interactions utilizing facial expression, gaze and speech-silence features. In [10] and [30], Xiao et al. analyzed therapists’ conversation to automatically classify empathic and non-empathic utterances using linguistic and acoustic information.

2.2.2 Research on Sentiment Analysis from Text

When it comes to the term “Sentiment Analysis” current state-of-art research are mostly focused on extracting it from textual information [31]. It could be from movie-reviews [32], tweets [33, 34], and newspaper articles and comments [35]. Sentiment analysis from text has been mostly used in terms of positive and negative polarity [36, 37, 38, 39, 34]. More detailed dimensions are proven to be very useful. For example, moods such as tension, depression, anger, vigor, fatigue, and confusion in tweets have been found to be good predictors of stock market exchanges [33]. It has also been found that it is possible to predict anger, sadness, and joy from LiveJournal blogs [40]. A study also suggests that it is also possible to distinguish Twitter users who are likely to share content generating joy or amusement from the ones who are likely to share content generating sadness, anger or disappointment with an accuracy of around 61% [35].

A document contains different subjects/topics where each topic is associated with a polarity. In [41], Nasukawa and Yi present that semantic information and sentiment lexicon can capture such polarity associated with the topic. Pand and Lee [42] present an approach, which first detects the subjectivity portion of text using a graph-based method then use a supervised classifier to detect the polarity. Entity based polarity detection first detect (e.g., people, places, things) from text then assigns scores indicating positive or negative opinion toward that. The study of Godbole in [43], present a system that captures sentiment of news and blog entities. Lin and He [44] proposed a joint sentiment/topic model by using Latent Dirichlet Allocation (LDA) with a sentiment layer, which detects topics of a document and assigns sentiment polarity on the topics. In [45], Taboada et al. present a lexicon-based approach to assign a positive or negative label to a text that captures the text’s opinion towards its main subject matter. Maas et al. [46] proposed a probabilistic model of documents, which learns word representations and used LDA to learns topics and used logistic regression for the classification. The study of Kouloumpis et al. in [37] present the use of n-gram, parts-of-speech tags and some lexical features for classifying tweets with sentiment polarity. In [47], the authors present a hybrid framework for sentiment analysis that includes a knowledge-based system and a machine learning module. A research application SentiStrength [48] utilizes a different source of information to assign a sentiment score to a short text [49, 50]. Such information includes word-list of sentiment, idioms, emoticons, negating words, linguistic rules and sentiment polarity classification algorithms. Last, but not least, a survey of sentiment analysis can be found in [51].

3 The Modal Model of Emotions

Many psychologists have studied emotional episodes from the point of view of appraisal dimensions. Gross  [13] has provided evidence that concepts such as emergence — derivation from the expectations of relationships — and unfolding — sequences that persist over time — may help in explaining emotional events. It has been shown that temporal unfolding of emotions can be conceptualized and experimentally tested [52]. The modal model of emotions developed by Gross [13, 53] emphasizes the attentional and appraisal acts underlying the emotion-arousing process. In Figure 1, we provide the original schema of Gross model. The individuals’ core Attention-Appraisal processes (included in the box) are affected by the Situation that is defined objectively in terms of physical or virtual spaces and objects. The Situation compels the Attention of the individual; it triggers an Appraisal process and gives rise to coordinated and malleable Responses. It is important to note that this model is dynamic and the situation may be modified (directed arc from the Response to the Situation) by the actual value of the Response generated by the Attention-Appraisal process. The modal model of emotions provides a useful framework for describing the contextual dynamics of emotions within an affective scene([54]), since it decomposes the emotional process in terms of situation selection, attentional deployment, and situational modification. Gross’ informal model can provide insight in the process where the empathic responses may modify the initial emotional context.

For guiding the observers through their perception process we rely on the modal model of emotions. That framework may provide the observers with grounded descriptions of the unfolding of the emotions in the affective scene. The annotator will need to identify the emergence, appraisal, and effects of emotions felt by the call center agent in the ongoing (sub-)dialogs. In doing so, the annotator should also be able to perceive if, and to what extent, an empathic response may modify a situation where other emotions such as frustration or anger are expressed by the customer. In section 4.3 we describe how the modal model of emotions applies to our annotation model of empathy.

4 The Empathy Annotation Model

The goals of the annotation model for empathy is to provide empirical evidence of the reference definition and annotated signal samples to train a computational model. In the following sections we describe some challenges in annotating real-life stimuli (section 4.1), we report on the qualitative data analysis of the dialogue corpus (section 4.2), we introduce the operational definition of empathy (section 4.3) in the context of call center conversations, and we describe the annotation process (section 4.4) and the evaluation of the annotators’ decisions (section 4.5).

4.1 Annotation of Real-Life Spoken Conversations

The annotation unit is the stimulus presented to the observer (annotator) to perform a selection task over a decision space such as the set of emotional state tags. In general the annotator may be presented with images or speech segments (stimuli), and a set of emotional labels. The stimulus is defined in terms of the medium the emotion is being transmitted through and its content and context. The medium may be speech [55], image [56] or multimodal [57]. The content refers to the information encoded in the stimulus signal such as facial expression of anger or a speech utterance. The context of the stimulus is represented by the spatial or temporal signals neighboring the stimulus. Knowledge of the context may be crucial in interpreting the cause of emotion manifestations. For a speech stimulus in a conversation the context is represented by the preceding dialog turns [55]. The above description of the annotation unit does not assume a univocal relationship between the occurrence of an emotion and its corresponding expression.

Most research in affective computing has been focused on stimuli that are designed in advance and are artificially generated. Respective examples of such stimuli are sentences to be read and actors enacting affective scenes [58]. Another limitation of acted annotation tasks is that stimuli are context-free and speech utterances or images are annotated in isolation.

The limitations of determining the temporal boundaries of the annotation units have motivated researchers to investigate the process of continuous annotation [29, 27]. Yet, state-of-the-art complete continuous affective scene annotation techniques are highly demanding for observers and are not very effective in terms of inter-annotator agreement. In our work we address the task of defining and searching the annotation unit in non-acted, real-life spoken conversations. We describe below how the annotation model we propose exploits the conversation context for the perception and annotation of the empathic events.

4.2 Qualitative Data Analysis

We have performed qualitative analysis of the spoken conversation corpus to gain insights into the speech signal instantiations of emotions and empathy. The analysis was carried out over a corpus of human-human dyadic Italian call center conversations that will be described in Section 5. We analyzed one hundred conversations (more than $11$ hours), randomly selected from the corpus, in particular the dialog turns where the speech signal showed the emergence of emotions (e.g. frustration, anger) on the customer channel and empathy on the agent channel.

The outcome of the qualitative analysis has supported the view that emotionally relevant conversational segments are often characterized by significant transitions in the paralinguistic patterns or the occurrence of lexical cues. As expected, such variations may co-occur not only with emotionally-connoted words but also with functional parts of speech (POS) such as Adverbs and Interjections. Phrases and Verbs, as shown in Table 1, could also lexically support the expression of emotional states.

In Table 1, we present a dialog excerpt with annotations. The dialog excerpt is reported in the first column of the table, where C is the customer, and A is the agent. The situation is the following: C is calling because a payment is actually overdue: he is ashamed for not being able to pay immediately and his speech has plenty of hesitations. This causes an empathic response by A: that emerges from the intonation profile of A’s reply and from her lexical choices. In the second question of A’s turn, she uses the hortatory first person plural instead of the singular one. Also, the rhetorical structure of A’s turn, i.e., the use of questions instead of assertions, conveys her empathic attitude. The annotator identified the context, i.e., the preceding dialog turn, and perceived the intonation variation. The annotator could then mark the speech segment corresponding to the intonation unit starting with the word proviamo (let us try) as onset of the emotional process.

4.3 The Operational Definition of Empathy

Following the modal model of emotions and the annotation model discussed above, we first describe the context of the situation, the attention, the appraisal and the response components of the empathic process.

The context of the situation: In call center conversations, customers may call to ask for information or for help to resolve technical or accounting issues. Agents are supposed to be cooperative and empathic and they are trained for the task. However, variabilities in the agents’ or customers’ personalities, behaviors and random events lead to statistical variations in the emotional unfolding. The operational definition of empathy that may be applied to this context requires the annotators to be informed of the social context and task. They are trained to focus over sub-dialogues where the agent anticipates or proposes solutions and clarifications (attention), based on the understanding of the customer’s problem (appraisal). As a consequence, the acts of the agent may prevent or releave customer’s unpleasant feelings (response). Therefore, we operationally define empathy as a situation where an agent anticipates or views solutions and clarifications, based on the understanding of a customer’s problem or issue, that can relieve or prevent the customer’s unpleasant feelings.

The selection of the stimuli includes a continuous search of the speech segments preceding and following the perception of the outset of the empathic segment. The task of the annotator is to identify the context (left of the outset) and the target (right of the outset) empathy segment. The context is defined to be neutral with respect to the target empathic segment. This does not mean that emotions rather than empathy cannot be manifested by the speaker in the speech segment that the annotator identifies as context, but that those possibly occurring emotions are not recognized as empathy by the annotator. The reference to a neutral, i.e., non-empathic, segment supports the annotators in their perception process while identifying the outset of the empathic segment. This annotation protocol was applied to annotate sequences of basic emotions occurring in affective scenes ([59]).

Instructions were given to the annotators to achieve the most confident decision in identifying and tagging the empathy segments, based on the perceived paralinguistic and/or linguistic cues. Based on prior research [60], we advised them to focus their attention on the variations in the paralinguistic cues (e.g., pitch rise or fall). The annotation guidelines provided examples like the ones described in section 4.2.

4.4 The Annotation Procedure

The typical corpus annotation process starts with the selection of a set of conversations and then the annotation of the speech segments in each conversation. In this work, we focused on first instances of empathy segments within each conversation. The rationale behind this choice was to maximize the number of annotated conversations with many speakers given limited resources. Annotators manually refined the boundaries of the segments generated by an off-the-shelf Speech/Non-Speech segmenter. The annotators tagged the Empathy segments on the agent channel and the basic emotions (Frustation and Anger) on the customer channel. The reason of choosing these basic emotions was due to the limited resources for annotation. Another reason was that we observed these emotions most frequently occurred in our corpus. Annotators were instructed to select the candidate segment pairs with a decision refinement process. Once the relevant speech region was identified, the annotators could listen to the speech as many times as they needed to judge if the selected segment(s) could be tagged with any of the target labels. The average per-conversation annotation time was 18 minutes for an average duration of a conversation of 6 minutes. Two annotators completed the annotation task. They used the EXMARaLDA Partitur Editor [61]. The corpus has been annotated with four labels: Empathy, Anger, Frustration, Neutral to describe the complete affective scene of the context (more in Section 5).

4.5 Evaluation of the Annotation Model

To assess the reliability of the annotation model, we designed the following evaluation task. Two annotators, with psychology background, worked independently over a set of 64 spoken conversations (approximately $7$ hours of speech) randomly selected from the call center corpus. The annotators were of similar age, same ethnicity and opposite gender. We intended to assess if the annotators could perceive a change in the emotional state at the same signal position (Neutral leading into Empathy), as well as their agreement with the assignment of the empathy label. In case of disagreement between the annotators a consensus meeting tried to reach a consensus. The inter-annotator agreement for the annotated segments between the two annotators is $53.1\%$, where the agreement on the same onset position is $31.2\%$. Annotating in continuous time space is typically a difficult task. Moreover, when it comes to annotate emotional aspects it became much more difficult due to the perception of the annotators in defining the emotional segment boundary in time space. This difficulty results in the low agreement of emotional onset boundary detection, which is reasonable.

To measure the reliability of the annotations we calculated inter-annotator agreement by using the kappa statistics [62]. It is frequently used to assess the degree of agreement among any number of annotators by excluding the probability that they agree by chance. The kappa coefficient ranges between 0 (agreement is due to chance) and 1 (perfect agreement). Values above $0.6$ suggest acceptable agreement ([63] [64],[65]). Our annotation task was challenging because it combined categorical annotation with continuous perception of the slowly varying emotion expression from speech-only stimuli. Thus we evaluated the agreement between annotators based on a partial match: two annotators agreed on the selection of the onset time stamps within a tolerance window of 5 sec. We found reliable results with kappa value for empathy $0.74$.

Most categorical emotion annotation research in speech deals with lower human agreement (greater than $0.50$) maybe due to a variety of factors, including short audio clips or utterance ([66, 55]), multi-label annotation tasks, and annotator agreement when the annotation task is based on continuous and discrete label annotations [29]. In our case the positive evaluation results may be motivated by the operationalized definition of empathy, by the observability of the complete paralinguistic and linguistic contexts, and finally by the binary annotation task.

5 Corpus Analysis

In this Section, we describe the spoken conversation corpus and its annotation statistics. We then report a correlation study and discover speech correlates in the speech signal (Section 5.2) and lexical features ( 5.3 ). These correlates are then used to guide the training of the automatic empathy classification model.

5.1 Corpus Description

The corpus includes $1894$ randomly selected customer-agent conversations, which were collected over the course of six-months, amounting to $210$ hours of speech data. These conversations were recorded on two separate channels of $16$ bits per sample and $8kHz$ sampling rate. The average length of the conversations was $406$ seconds.

We analyzed the distribution of emotion state label sequences describing the affective scene in the conversations. In Figure 2, we present an example of an emotional sequence, which manifests the affective scene. We analyzed such sequential emotional patterns in the corpus to understand the insights of the agent and customer affective behavioral aspects. More details of such findings can be found in [67]. We observed that empathy was perceived by annotators in $27.72\%$ of the conversations out of the whole set as can be seen in Table 2, in which column represents agent’s channel and row represents customer’s channel. By analyzing only empathic conversations ($27.72\%$), we observed that when agent manifested empathy no frustration or anger was manifested by the customer in $70\%$ of the cases. This affective scene may be explained, in part, by specific training of agents in anticipating customers’ issues and taking appropriate actions. For the remaining $30\%$ of the conversations customers had manifested anger or frustration or both.

Moreover, for $12.99\%$ (see in Table 2) of the conversations out of whole set, customers had manifested anger or frustration or both while no empathy had been perceived on the agent side. From our analysis we can then characterize the stereotypical situations that occur in the affective scene with the two following scenarios:

Customer shows anger and/or frustration and the agent does not react to the speech and language cues of the customer. This is the case in which agent was expected to be empathic, but failed to recognize or react to customer’s signals.

• 2.

Agent is empathic in response to or in anticipation of the customer’s anger, frustration or both.

5.2 Acoustic Feature Analysis

We investigated and compared the pattern sequences of low-level acoustic features before and after the onset point, from the neutral to the empathy segment. An example of the speech segment annotation is shown in Figure 3, where we plot the spectral centroid222Spectral centroid measure the brightness (i.e., high frequency signals) of a sound. It is defined as the frequency-weighted sum of the power spectrum normalized by its unweighted sum [68]. Using spectral centroid we can capture how the centroid of two signals differ. feature values across the neutral and empathy connoted segment. Each segment is 15 seconds long and the onset is marked by a vertical bold line, which separates the left (context) and right segment annotated with empathy. From the signal trend of this feature we see that there is a distinctive profile change, which is corroborated by its high statistical significance ($p-value=4.61E-51$, $d=1.4$ and $t=-16.6$ using two-tailed two-sample t-test).

The low-level features were extracted from both left and right segment with 100 overlapping frames per second, pre-emphasis with k=0.97, and hamming-windowing, using openSMILE [69]. For voice quality features we used gaussian windowing function. Then, we computed averages for each segment of the corresponding conversation. In order to evaluate the relevance of each feature we applied a statistical significance test, the two-tailed two-sample t-test at p-value $=0.01$. We analyzed $45$ low-level acoustic features from five categories: pitch ($4$), loudness ($1$), zero-crossings (1), spectral ($13$), and auditory-spectrum bands (26). In Table 3, we list the acoustic features that passed the significance test for male ($40$ features) and female ($34$ features) speakers. We also report effect sizes, d, which are computed using Cohen’s d, as in Equation 1. Effect size refers to how big or small the difference in means of two groups. Here two groups refers to the neutral and empathy segments. The number of samples for this analysis is $302$ conversations, $n=302$, sample size.

[TABLE]

where $m1$ and $m2$ are the means of two samples, and $\sigma_{1}$ and $\sigma_{2}$ are the standard deviations of the two samples.

From our analysis we observe that the pitch patterns are higher in non-empathic segments. The spectral features such as centroid and flux are more stable and smooth when the agent is empathic compared to abrupt changes in non-empatic segments. Spectral patterns captures the perceptual impression of sharpness of sounds. The vocal pattern of loudness is higher in non-empathic situations while it is low when the agent is empathic. We also observed that the auditory-spectrum bands of non-empathic segments were comparatively higher. Our analysis on the relevance of pitch and loudness for empathy signal realization is consistent with the findings of [70] and [30]. There are no significant differences in the relevance of features in Table 3 for male and female speakers.

The purpose of this analysis was to understand which type of feature could be used for designing classification model. We later observed that our analysis corroborates with the finding of the feature selection procedures in classification experiments.

5.3 Lexical Feature Analysis

Several categories of personnel who interact with customers or patients, including call center agents and physicians, are trained to improve their communication skills and develop empathy in their interactions through careful choice of words [72, 73]. For example, they are recommended to use phrases such as “I understand” when they are listening to the customer who is explaining their problem, or to use “I would”. For example, after hearing a customer’s story, agent may respond by saying “I would be upset as well if I were in a similar situation”, before proceeding to propose possible solutions or provide advice.

In our call center corpus, we analyzed the lexical realization occuring in empathic and neutral segments by comparing the different word frequencies and POS distributions of unigrams, bigrams and trigrams respectively. The goal was to extract the most relevant word $n$-grams from empathic as well as neutral segments.

We tested the statistical significance over the observed differences with a two-tailed two-sample t-test and a p-value of $0.01$ with the same number of conversations ($n=302$) we used for the analysis of acoustic features.

The comparison between neutral and empathy word trigrams showed that agents’ phrases such as vediamo un po’ (let’s see), vediamo un attimo (let’s see a bit), vediamo subito allora (let’s see now, then) are statistically significant lexical cues for the agents while interacting and manifesting their empathy. It is worth noticing that the Italian verb vedere (see) is more frequently used in the first person plural; the same holds true for other frequent verbs such as facciamo (let’s do) and controlliamo (let’s check). Those lexical choices are usual when the speaker cares about the problem of the other person. Similarly, significantly different rankings in the empathic distribution affect unigrams, bigrams and trigrams such as non si preoccupi (do not worry), allora vediamo (so, let’s see) that are often used in Italian to grab the floor of the conversation and reassure the other person. For the above lexical features p-value was $<0.01$ and the range of effect sizes, d, was $[0.7~{}to~{}3.8]$, and t-statistic was $[-5.5~{}to~{}-3.0]$ as shown in Table 4.

Regarding the POS distributions, the Adverbs that occur frequently in the empathy distribution, such as assolutamente (absolutely) and perfettamente (perfectly), may have a kind of evocative potential for showing understanding of the other person’s point of view, in particular when they are uttered with a tone of voice appropriate to the context.

6 Automatic Classification of Empathy

In this Section, we describe the training of an automatic classification system for the recognition of empathy from spoken conversations. We report experimental details of the feature extraction, fusion and classification task and discuss the results.

6.1 Classification Task and Data-set

In the automatic classification experiment, our goal was to investigate the empathic conversational segments of the operator. We have selected a subset of the corpus, that includes a total of 526 conversations annotated with automatic speech transcriptions as well as as Neutral, Empathy segment labels. This subset of the corpus allows us to perform a complete computational model training and evaluation in noisy and clean input signal conditions. We partitioned the data-set into train, development and test with 70%, 15% and 15% partitions and no-speaker overlap amongst them. In order to train and evaluate the system we extracted neutral-empathic segment-pair from these conversations, which has been obtained from manual annotation.

Duration-based descriptive statistics of these segment pairs are provided in Table 5 along with averages and standard deviations on the natural distribution of the data. The segment length of neutral is comparatively longer than the empathic segment as we see in the Table, since it spans from the start of the conversation until the onset of the first empathic event. The total net duration of these segment-pairs is $\sim$35 hours. It is quite usual that in real-world conversations, the distribution of neutral segments is significantly higher than the manifested emotions as can also be seen in [74].

6.2 Classification System

In Figure 4, we present a computational architecture of the automatic classification system, which takes the agent’s speech channel as input, then pass it to the automatic speech vs non-speech segmenter. After that it generates a binary decision for each speech segment of the agent’s behavior in terms of neutral vs empathy. In order to evaluate the relative impact of lexical features we considered the case of noisy transcriptions (left branch in Figure 4) provided by an automatic speech recognizer (ASR). We extracted, combined, and selected acoustic features directly from the speech signal and generated the classifier’s training set. We implemented both feature fusion and decision fusion algorithms (bottom part of Figure 4) to investigate the performance of different classifier configurations. This architectural design can be used in real time application, which combines all automatic processes.

6.3 Speech vs Non-Speech Segmenter

An HMM-based speech vs non-speech segmenter has been trained using a set of $150$ conversations, containing approximately $100$ hours of spoken content and used Kaldi [75] for the training and decoding processes. Training data has been prepared using force-aligned transcriptions. Mel Frequency Cepstral Coefficient (MFCC) and their derivatives has been used as features. Number of gaussian and beam width have been optimized using a development set of $50$ conversations. Final model has been tested using a test of $50$ conversations. The F-measure of the system was $66.0\%$ on the test set.

6.4 Undersampling and Oversampling

The statistics in Table 5 manifest the data imbalance problem for the two classes, Empathy and Neutral. Once the manual segments are processed through the automatic segmenter, the ratio of Empathy/Neutral labeled segments is $6\%$ vs $94\%$. A common approach to cope with that is via oversampling or undersampling in the data or feature space. We have under-sampled the instance of the majority class (i.e., neutral) at the data preparation phase and oversampled the minority class (i.e., empathy) after the feature extraction phase. In the literature it is reported that the combination of oversampling and undersampling often leads to better performance [76].

The idea of our undersampling is a novel approach, in which we randomly pick samples from the majority class. At the same time, we maintain a variation of the random samples. The intuition behind is that classifier does not learn much if it basically sees many similar samples. Our way of picking random samples maintains a variation and upon feeding such samples for the same class classifier learn better predictors. We maintain this variation by defining a set of bins with different segment lengths, and then randomly select $N$ segments from each bin. We used $N=1$ for this study. The number of bin and size of $N$ has been optimized empirically on the development set and by investigating the descriptive statistics such as percentiles, mean and standard deviation. The pseudocode of the algorithm is presented in Alam et el. work [67]. The implementation of this algorithm will be made publicly available. The undersampling stage generated a $18\%$ *vs * $82\%$ ratio of Empathy *vs * Neutral segments up from the initial $6\%$ vs $94\%$.

For oversampling we used Synthetic Minority Oversampling Technique (SMOTE) [76] and its open-source implementation in Weka [77]. In SMOTE, the upsampled examples (i.e., empathy) are generated based on the $K$ nearest neighbors of the minority class. Nearest neighbors have been chosen randomly based on the percentage of target oversampling. It computes the difference between the feature vector and its nearest neighbor. Then, multiply this difference by a random number between 0 and 1 and add it to the feature vector. The oversampling was tuned on the development set and we achieved a further improvement on the imbalance problem. Before oversampling the class distribution was $18\%$ vs $82\%$, and following oversampling that it became $30\%$ vs $70\%$. Using this approach classifier learns the variations of segments in different lengths and also reducing the effect of imbalance class distribution.

6.5 Feature Extraction

In this section we report the algorithms used in the extraction of the acoustic, linguistic and psycholinguistic features in the empathy recognition task. We discuss differences and similarities of the features used in our experiments to those used in other emotion and personality recognition tasks [78, 79].

6.5.1 Acoustic Features

Recent studies in emotion and personality recognition showed that the paralinguistic properties of speech are well represented by low-level features [80, 78, 79]. We followed a similar approach and we extracted a very large number of low-level features and their statistical functionals using the openSMILE tool [69]. The low-level acoustic features include the feature set of the computational paralinguistic challenge’s (COMPARE-2013) feature set [81], Geneva minimalistic acoustic feature set [82] and few more formant features. For the sake of the replicability we made the configuration files of the feature set publicly available333https://github.com/firojalam/openSMILE-configuration. We extracted low-level acoustic features at approximately $100$ frames per second. Regarding voice-quality features the frame size was $60$ milliseconds with a Gaussian window function and $\sigma=0.4$. Regarding other low-level features the frame size was $25$ milliseconds with a hamming window function. The details of the low-level features and their statistical functional are provided in Table 6. After feature extraction, the size of the resulted feature set is $6861$.

6.5.2 Lexical Features

We extracted lexical features from automatic transcriptions. The automatic transcriptions were generated using a large vocabulary ASR system [83]. We designed HMM-based ASR system using a subset of $1894$ conversations containing approximately $100$ hours of spoken content and a lexicon of size $\sim$15K words. From the conversations we extracted Mel Frequency Cepstral Coefficient (MFCC) features and then spliced by taking 3 frames from each side of the current frame. It was followed by Linear Discriminant Analysis (LDA) and Maximum Likelihood Linear Transform (MLLT) feature-space transformations to reduce the feature space. Then, we trained acoustic model using speaker adaptive training (SAT). In order to achieve the best accuracy we also used Maximum Mutual Information (MMI). The Word Error Rate (WER) of the ASR system was $31.78\%$ on the test set and $20.87\%$ on the training set, using a trigram language model of perplexity $87.69$. For the training and decoding process, we used Kaldi [75].

The training set of the ASR system overlapped with the task classification test corpus that was used in the classification experiments (Section 6.7). However, the training error rate of $20.87\%$ was still realistic and useful for the classification task.

We mapped the speech transcriptions into lexical features in the step preceding the training of the classifier. The transcriptions of each segment were converted to bag-of-words vectors weighted with logarithmic term frequencies (tf) multiplied with inverse document frequencies (idf), presented in equation 2.

[TABLE]

where $f_{ij}$ is the frequency for word $i$ in conversation $j$. In order to take advantage of the contextual benefits of $n$-grams, we extracted trigram features. Because this resulted in an unreasonably large dictionary and we filtered out lower frequency features by preserving 10K most frequent $n$-grams.

6.5.3 Psycholinguistic Features

Similar to the lexical features we extracted the so-called psycholinguistic features from automatic transcriptions. Over the past few decades, Pennebaker et al. have designed psycholinguistic word categories using high frequency words and developed the Linguistic Inquiry Word Count (LIWC) [84]. These word categories are mostly used to study gender, age, personality, and health to estimate the correlation between these attributes and word usage (see [79, 59] and the references therein). The types of LIWC features include the following:

General features such as word count, average number of words per sentence, percentage of words found in the dictionary and percentage of words longer than six letters and numerals. 2. 2.

Linguistic features include pronouns and articles. 3. 3.

Psychological features include affect, cognition and biological phenomena. 4. 4.

Features about personal concern include work and home. 5. 5.

Paralinguistic features include accents, fillers and disfluencies. 6. 6.

Punctuation categories include periods and commas.

We used the dictionary that is available within LIWC for Italian [85]. The Italian dictionary contains $85$ word categories. LIWC extracted $5$ general descriptors and 12 punctuation categories constituting a total of $102$ features. The LIWC feature processing differs according to types of features. Some features are counts and others are relative frequencies (see [86]).

6.6 Feature Fusion and Selection

We applied feature selection to each individual feature set (Figure 4). We also merged acoustic and lexical features into a single vector to represent each instance in a high-dimensional feature space. Let $A=\left\{a_{1},a_{2},...,a_{m}\right\}$ and $L=\left\{l_{1},l_{2},...,l_{n}\right\}$ denote the acoustic and lexical feature vectors respectively. The feature-combined vector was $Z=\left\{a_{1},a_{2},...,a_{m},l_{1},l_{2},...,l_{n}\right\}$ with $Z\in R^{m+n}$. Given the high-dimension of the feature vector and the curse of dimensionality, we applied feature selection on the $Z$ space to achieve an optimal feature dimension of size $k$, $k<\left(m+n\right)$. Another objective of the feature selection process was to find the best compromise between the dimension of the input and the performance of a target classifier.

For feature selection we used the Relief [87] feature selection technique. In [2], we comparatively evaluated the Relief method against other algorithms and it outperformed them in classification performance and computational cost. We ranked the feature set according to Relief scores and generated learning curves by incrementally adding batches of ranked features. We then selected the optimal set of features by stopping when performance saturated or started decreasing [2].

The goal of Relief is to estimate weights to find relevant attributes with the ability to differentiate between instances of different classes under the assumption that nearest instances of the same class have the same feature values and different class have different values. Relief estimates weight of a feature, $F$, using Equation 3.

[TABLE]

where $x$ is a value of $F$, nearest miss and nearest hit are the nearest instances of the same and a different class, respectively.

Since some feature selection algorithms do not support numeric feature values such as information gain and suffer from data sparseness such as Relief, we discretized feature values into $10$ equal-frequency bins [77] as a pre-processing step of feature selection. The equal-frequency binning approach divides data into $k=10$ groups, where each group contains approximately equal number of values. Moreover, the feature selection and discretization approach were performed on development set (see Section 6.7) in order to avoid biases in classification experiments. The equal-frequency binning approach and the size of the bin, $k=10$, were empirically found optimal in other paralinguistic classification task [2]. Furthermore, we did not apply the selection algorithm to the psycholinguistic features due to the limited size of the feature set.

After selecting the features, we analyzed the top-ranked features per category. The top ranked acoustic features included spectral, mfcc, probability of voicing, and pitch. The spectral feature type included spectral variance, flux, auditory-spectrum with bands $1K-4K$ Hz, roll-off points. The highly relevant LIWC feature category included verb, article, negations, and, social and cognitive processes. The lexical features included allora vediamo se (so let’s see if), assolutamente (absolutely), sicuramente (certainly) tranquillamente (nicely) and non si preoccupi (do not worry). The statistical analysis of acoustic and lexical features for the neutral-empathy segment annotation (see Section 5.2 and 5.3) corroborate these latter findings of the top-ranked features designed for the classification step.

6.7 Classification and Evaluation

6.7.1 Baseline

To compare the results of our proposed approach we have conducted an experiment to have baseline results. The typical approach of computing baseline to choose majority class [77] or random selection of class labels. For our study, we have chosen the latter approach, to compute the baseline, we have randomly selected the class labels based on the prior class distribution.

6.7.2 Proposed Classification Models

We designed our classification models using Support Vector Machines (SVMs). One important challenge in designing classification models using acoustic, lexical and combination of $acoustic+lexical$ features was their higher dimensional feature space compared to the size of the dataset. In order to deal with such a problem, we have chosen to use the linear kernel of the SVMs, which we found useful in other paralinguistic task [2]. For designing the classification model using psycholinguistic features we used the Gaussian kernel of the SVMs as the dimensionality of this feature set is small compared to other sets. For the experiment, we used Sequential Minimal Optimization (SMO) [88], which is designed to solve quadratic optimization problems of SVMs. This is available with an open-source implementation by Weka machine learning toolkit [77]. For the training, we optimized the penalty parameter $C$ of the error term by tuning it in the range $C\in[10^{-5},...,10]$ and the Gaussian kernel parameter $G$ in the same range as well, using the development set. To obtain the results on the test set we combined the training and development set and trained the models using the optimized parameters. We have not attempted to use a deep neural network, due to the smaller sized data-set.

Regarding the classifiers trained on different feature sets, we combined decisions from each classifier by applying majority voting, as in Equation 4.

[TABLE]

where $H(s)$ is the combined classification decision for an input instance $s$; $h_{i}^{j}(s)$ is the output of the classifier $h_{i}$ for the class label $c_{j}$; $i=1...T$ is the number of classifiers; $j=1...C$ is the number of classes.

In the experiments we used SMO across different classification systems and evaluated different sets of features such as lexical vs acoustic.

6.7.3 Evaluation

Evaluation at the segment level poses a great challenge. The evaluation procedure requires an alignment of the manual segment and labels with the output of the automatic segmentation and classification system described in the previous section. In Figure 5, we present an alignment example of the reference (manual) and automatically generated segments and their labels. The reference segmentation spans from $t=0$ to $t=t_{e}$ and labels the Neutral (N) segment spanning from $t=0$ to $t=t_{i}$ and the Empathy (E) segment from $t=t_{i}$ to $t=t_{e}$. Automatic segments inherit the reference label that falls inside its boundaries (e.g., the segment spanning the interval [$0,t_{1}$] or [$t_{3},t_{4}$]). For the evaluation purpose, automatic segments that span across the onset, $t=t_{i}$, and end, $t=t_{e}$, (e.g., the segment spanning the interval [$t_{2},t_{3}$]) are split in two segments with two distinct reference labels. For instance the segment spanning [$t_{2},t_{3}$] will be evaluated with the segment [$t_{2},t_{i}$] (reference label (N)) and the segment [$t_{i},t_{3}$] (reference label (E)). The alignment process will generate the correct label statistics for all segments as shown on the last row of Figure 5.

We have measured the performance of the system using the Un-weighted Average (UA), which has been widely used in the evaluation of paralinguistic tasks [3]. We have extended such measure to account for the segmentation errors as evaluated in similar cases by NIST in diarization tasks [89, 90]. UA is the average recall of positive and negative classes and is computed as $UA=\frac{1}{2}\left(\frac{tp}{tp+fn}+\frac{tn}{tn+fp}\right)$, where $tp$, $tn$, $fp$, $fn$ are the number of true positives, true negatives, false positives and false negatives, respectively. We computed $tp$, $tn$, $fp$ and $fn$ from a weighted confusion matrix, as shown in Equation 5, where the weight for each instance is the corresponding segment length:

[TABLE]

In Equation 5, $C(f)$ is the $2\times 2$ confusion matrix of the classifier $f$, $s$ is the automatic segment in the test set $S_{T}$, including the segment with boundary $t_{i}$ and $t_{e}$ (see Figure 5), $length(s)$ is the duration of $s$, $y$ is the reference label of $s$, $f(s)$ is the automatic label for $s$. The indices $i$ and $j$ represent the reference and automatic class label of the confusion matrix.

6.8 Classification Results

In Table 8, we report the performances of the automatic classification system trained on different feature types: lexical (automatic transcriptions), acoustic and psycholinguistic. We report test set results for feature combination-based system as well as and classifier combination. In the latter system we applied majority voting. For the statistical significance, we have computed McNemar’s significant test over the test set [91].

For single feature-type systems, acoustic-based models provided the best performance compared to lexical and psycholinguistic alone. The results of acoustic-based system are significantly better than random baseline with $p<2.2E-16$. The acoustic-based system provides a useful and low-computation classification model, when no automatic transcriptions are available. LIWC’s system performance improve over the lexical-only system with very few lexical features ($89$). In addition, all system’s UAs are higher and statistically significant compared to the baseline results with $p<2.2E-16$, chi-square value $228.6~{}to~{}1866.9$ and d $0.4~{}to~{}1.1$. For McNemar test, the test statistic is usually computed by a paired version of Chi-square and effect size, d, is computed using Cramer’s $\phi$. The value of $\phi$, $0.1$, $0.3$ and $0.5$ represents small, medium and large effect respectively.

In terms of feature and system combination, we obtained the best results with majority voting. The statistical significance test showed that the results of the majority voting are statistically significant compared to any other system’s results with $p<=0.0004$, chi-square value $12.2~{}to~{}1446.5$ and d $0.1~{}to~{}1.0$. Compared to the baseline, the best model for automatic classification provides a relative improvement over the baseline of $35.7\%$. Linear combination of lexical with acoustic features has not improved performance, despite its success in other paralinguistic tasks [79]. It has not even improved performance when combined with feature selection.

In order to assess the impact of the automated transcriptions, in a different study, we compared the performance between automatic and manual transcriptions for a automatic classification of emotional state. The results show that performance differences are very low, only $1.2\%$ drop with automated transcriptions [92]. Therefore, we found that the use of automatic transcriptions are reasonable for the experiment given that manual transcriptions are not available in all cases.

6.9 Discussion

The ability to detect empathy is very useful in all those human tasks that are geared towards the assessment of human interactions (e.g., teacher-student, doctor-patient, customer-operator etc.).

This paper presents the results of a quantitative study of empathy in spoken conversations. In order to obtain such results, we annotated a corpus of real human-human spoken conversations by using an annotation framework that was defined for this work and based on the psychological ‘modal model of emotions’ by Gross. The goal of the human annotators was to label the speech segments where they could perceive a possible expression of empathy. Their perception task was guided by the operational definition of empathy that was based on the ‘modal model’ theory. The novelty of our emotion annotation framework is related to the fact that it provides the annotators with operational definitions of the target emotions (in this case empathy) that takes into consideration the situational context where the emotions unfold.

This annotation task was done on a continuous time scale: this is a difficult task due to the variability of emotional expressions and to the individual nature of the perceptual process of human beings. Also, the identification of the emotional segment boundary is a difficult task, and reaching an agreement on the same boundary position in continuous time scale is a further challenge. Providing the annotators with operational definitions of emotions, such the one we provided for empathy in the present work, was useful for constraining those individual variability factors, and the results obtained by evaluating the inter-annotator agreement may support this view.

We have extensively analyzed acoustic and lexical features to understand their correlations with empathy. We observed that certain features are highly correlated with empathy. For designing the automatic classification system, we used acoustic, lexical and psycholinguistic features. From the classification results in Section 6.8, we observe that the results of our proposed classification model perform better than random baseline, which indicates that this results could be used as a starting point. In the comparison of different feature sets, we obtained higher results using acoustic features, and it would be useful when no transcription available. Lexical features rely on ASR system and improvement could be possible with a higher accuracy of the ASR system. LIWC features extracted using transcriptions are based on the lexical mapping, which can also be improved by enriching the lexical knowledge. One possible approach of enriching lexical knowledge is by using Wordnet or finding similar words from word-embedding vector [93].

The motivation of the study of feature level combination was that emotional aspects are represented in both verbal and vocal-nonverbal acoustic space and upon combining them in linear space can provide better results. However, the challenge here is that the vector space representations are different such as dense representation in acoustic space and sparse representation of lexical space. Future studies might need to focus finding a better combination strategy. One possible approach would be transforming the sparse lexical representation into dense representation either using word-embedding vector or other feature transformation approaches such as principal component analysis.

The present work suggests that it is possible to detect empathy. From our analysis, we observed that there are spoken language features that correlate well with empathy. The pitch and loudness are amongst the most correlated acoustic features ( see table 3 ) with empathy. The lexical features also play an important role in signaling empathy to the conversation participant and we have shown statistical significant patterns from the Italian corpus (see table 4.

There are aspects of the empathy that are very relevant and could benefit from the annotation framework and models presented in this paper. In particular the aspect of the coordination of emotions (e.g., anger or frustration) on one side (e.g., customer) and empathy on the other side (e.g., agent side). We have focused on analyzing behavioral cues both in terms of paralinguistic and linguistic cues, which would be useful in designing empathic virtual agent. For example, lexical content “let us see, perfect” with medium pitch voice could be useful cues to represent empathic response.These aspects are very important if we need to design predictive models for human-machine conversations or automatically evaluate the interactions of individuals.

One of the important research challenges is the variability of empathy in diverse affective scenes, domains, cultures, and languages. We believe the annotation model and classification system are robust enough to be extended or adapted to such different contexts to create new baselines and work on adaptation models such as [daume2009frustratingly, 94, 95] to further enhance the model.

Another important contribution of this paper is the automatic classification system of empathy in spoken conversations. Its design and classification results indicate that it is possible to support in the future a controlled-model in a empathic human-computer dialogue system. For human-machine scenario, further study is needed to understand that the coordination between a virtual empathic agent and a human can achieve a stable equilibrium. The analysis we have carried out maybe a starting point to train this coordination model, however further human-machine experimental studies will be needed.

7 Conclusion

Empathy refers to an emotional state triggered by a shared emotional experience. In this paper, we have addressed the problem of observing and annotating instances of empathy in real-life spoken conversations. The annotation process describes the scene through the Situation-Attention-Appraisal-Response model. We have operationalized the definition of empathy and designed an annotation process of the human-human dialogues in call centers. We have designed and evaluated a system that automatically segments and classifies empathic events from spoken conversations. We have investigated the effectiveness of acoustic, lexical, and psycholinguistic features and their combinations. The performance of our best classification system is very promising and significantly above the random baseline. The annotation model and classifier performances may lead to extensions in other situations and applications in human-machine interaction and automatic behavior analysis. In future work, we foresee to extend this work with a comparative study of the concurrent signals from agent and operator channels and understand the effect of interlocutor’s behaviors on each other. The empathic behavioral patterns might be different for human-machine interaction scenarios, however, we believe our findings would be helpful to study such interactions in future.

Acknowledgment

The research leading to these results has received funding from the European Union - Seventh Framework Programme (FP7/2007-2013) under grant agreement n° 610916- SENSEI.