Crowdsourcing and Validating Event-focused Emotion Corpora for German and English

TL;DR

This paper introduces deISEAR, a German emotion corpus created through crowdsourcing, and demonstrates that English-to-German transfer learning for emotion classification maintains performance.

Contribution

It develops the first German emotion corpus aligned with ISEAR and shows effective cross-lingual transfer using machine translation.

Findings

deISEAR corpus created for German emotion analysis

Cross-lingual transfer from English to German maintains classification performance

Crowdsourcing effectively gathers emotion descriptions and annotations

Abstract

Sentiment analysis has a range of corpora available across multiple languages. For emotion analysis, the situation is more limited, which hinders potential research on cross-lingual modeling and the development of predictive models for other languages. In this paper, we fill this gap for German by constructing deISEAR, a corpus designed in analogy to the well-established English ISEAR emotion dataset. Motivated by Scherer's appraisal theory, we implement a crowdsourcing experiment which consists of two steps. In step 1, participants create descriptions of emotional events for a given emotion. In step 2, five annotators assess the emotion expressed by the texts. We show that transferring an emotion classification model from the original English ISEAR to the German crowdsourced deISEAR via machine translation does not, on average, cause a performance drop.

Tables

Table 1. Table 1: Statistics for prompting emotions across the average number of tokens (#tok) and the extra-linguistic labels of the descriptions. Temporal Distance, Intensity and Duration report the number of descriptions for events which took place days (D), weeks (W), months (M) or years (Y) ago, which caused an emotion of a specific intensity (NV: not very intense, M: moderate, I: intense, VI: very intense) and duration (min: a few minutes, one hour: h, multiple hours: > > h, one or multiple days ≥ \geq d); Gender counts of the annotators are reported in the last column (male: M, female: F, other: O).

		Statistics		Temporal Distance					Intensity					Duration					Gender
	Emotion	#tok		D	W	M	Y		NV	M	I	VI		min	h	$>$h	$\ge$d		M	F	O
	Anger	15.1		46	25	31	41		3	25	67	48		23	29	39	52		112	31	–
	Disgust	13.1		38	38	42	25		12	52	48	31		95	37	8	3		110	33	–
	Fear	14.0		25	32	37	49		4	24	58	57		50	32	31	30		109	34	–
	Guilt	13.8		36	27	30	50		8	57	54	24		41	29	43	30		116	27	–
	Joy	11.6		40	30	29	44		2	18	60	63		14	18	42	69		107	35	1
German	Sadness	11.5		29	26	42	46		3	31	43	66		16	9	27	91		113	30	–
	Shame	13.2		25	28	36	54		24	56	41	22		72	28	24	19		116	27	–
	Sum	13.2		239	206	247	309		56	263	371	311		311	182	214	294		783	217	1
	Anger	28.3		45	29	25	44		9	34	48	52		30	23	36	54		62	81	–
	Disgust	22.4		57	25	21	40		12	51	37	43		66	27	24	26		57	86	–
	Fear	27.0		19	29	36	59		2	30	57	54		52	29	35	27		66	77	–
	Guilt	25.5		33	24	27	59		25	52	43	23		26	39	28	50		59	84	–
English	Joy	23.6		32	24	31	56		2	27	48	66		14	13	43	73		60	83	–
	Sadness	21.6		40	24	31	48		10	45	38	50		17	21	23	82		62	81	–
	Shame	24.8		21	22	19	81		16	51	42	34		29	25	39	50		57	86	–
	Sum	24.7		247	177	190	387		76	290	313	322		234	177	228	362		423	578	–

Table 2. Table 2: Number of descriptions whose prompting label (column Emotion) agrees with the emotion labeled by all Phase-2 annotators ( = = 5), by at least four ( ≥ \geq 4), at least three ( ≥ \geq 3), at least two ( ≥ \geq 2), at least one ( ≥ \geq 1).

	German						English
Emotion	$\ge$1	$\ge$2	$\ge$3	$\ge$4	$=$5		$\ge$1	$\ge$2	$\ge$3	$\ge$4	$=$5
Anger	135	125	107	81	52		137	129	112	89	59
Disgust	139	134	130	124	91		118	101	84	76	53
Fear	134	124	108	99	78		136	131	124	116	86
Guilt	137	126	102	67	31		137	130	124	89	44
Joy	142	142	142	140	136		143	143	143	143	137
Sadness	132	123	113	97	76		140	133	131	116	97
Shame	128	109	86	66	41		116	92	64	41	23
Sum	947	883	788	674	505		927	859	782	670	499

Table 3. Table 3: Event type analysis: Cells are counts of post-annotation out of 55 descriptions for each emotion.

	Dimension	Anger	Disgust	Fear	Guilt	Joy	Sadness	Shame
German	General event	4	2	1	0	0	1	0
	Future event	0	0	1	0	0	0	0
	Past event	51	53	53	55	55	54	55
	Prospective	1	0	4	0	1	1	0
	Social	30	28	24	29	24	40	25
	Self conseq.	37	34	37	26	44	21	37
	Conseq. oth.	21	9	19	34	16	34	14
	Situat. control	2	5	4	24	9	3	19
	Responsible	20	31	17	51	26	23	40
English	General event	2	2	2	2	0	3	0
	Future event	0	0	0	0	0	0	0
	Past event	53	53	53	53	55	52	55
	Prospective	0	0	14	0	1	0	0
	Social	50	37	30	41	39	49	41
	Self conseq.	29	26	42	20	35	16	32
	Conseq. oth.	29	23	19	34	24	43	29
	Situat. control	3	7	8	31	15	2	24
	Responsible	13	29	34	53	34	16	43

Table 4. Table 4: Performance of ISEAR-trained classifier on our crowdsourced corpora, per emotion and micro-average F 1 subscript F 1 \text{F}_{1} ( μ 𝜇 \mu F 1 subscript F 1 \text{F}_{1} ).

Dataset	$\mu$${\text{F}}_1$	An	Di	Fe	Gu	Jo	Sa	Sh
deISEAR	47	29	49	48	42	68	53	39
enISEAR	47	27	45	57	41	67	58	32

Full text

1 Introduction

Feeling emotions is a central part of the “human condition” (Russell, 1945). While existing studies on automatic recognition of emotions in text have achieved promising results (Pool and Nissim (2016); Mohammad (2011), i.a.), we see two main shortcomings. First, there is shortage of resources for non-English languages, with few exceptions, like Chinese Li et al. (2017); Odbal and Wang (2014); Yuan et al. (2002). This hampers the data-driven modeling of emotion recognition that has unfolded, e.g., for the related task of sentiment analysis. Second, emotions can be expressed in language with a wide variety of linguistic devices, from direct mentions (e.g., “I’m angry”) to evocative images (e.g.,“He was petrified”) or prosody. Computational emotion recognition on English has mostly focused on explicit emotion expressions. Often, however, emotions are merely inferable from world knowledge and experience. For instance, ”I finally found love” presumably depicts a joyful circumstance, while fear probably ensued when ”She heard a sinister sound”. Attention to such event-related emotions is arguably important for wide-coverage emotion recognition and has motivated shared tasks Klinger et al. (2018), structured resources Balahur et al. (2011) and dedicated studies such as the “International Survey on Emotion Antecedents and Reactions” (ISEAR, Scherer and Wallbott, 1994). ISEAR, as one outcome, provides a corpus of English descriptions of emotional events for 7 emotions (anger, disgust, fear, guilt, joy, shame, sadness). Informants were asked in a classroom setting to describe emotional situations they experienced. This focus on private perspectives on events sets ISEAR apart. Even though from psychology, it is now established in natural language processing as a textual source of emotional events.

With this paper, we publish and analyze deISEAR, a German corpus of emotional event descriptions, and its English companion enISEAR, each containing 1001 instances. We move beyond the original ISEAR in two respects. (i), we move from on-site annotation to a two-step crowdsourcing procedure involving description generation and intersubjective interpretation; (ii), we analyze cross-lingual differences including a modelling experiment. Our corpus, available at https://www.ims.uni-stuttgart.de/data/emotion, supports the development of emotion classification models in German and English including multilingual aspects.

2 Previous Work

For the related but structurally simpler task of sentiment analysis, resources have been created in many languages. For German, this includes dictionaries (Ruppenhofer et al., 2017, i.a.), corpora of newspaper comments Schabus et al. (2017) and reviews Klinger and Cimiano (2014); Ruppenhofer et al. (2014); Boland et al. (2013). Nevertheless, the resource situation leaves much to be desired. The situation is even more difficult for emotion analysis. Emotion annotation is slower and more subjective (Schuff et al., 2017). Further, there is less agreement on the set of classes to use, stemming from alternative psychological theories. These include, e.g., discrete classes vs. multiple continuous dimensions (Buechel and Hahn, 2016). Resources developed by one strand of research can be unusable for the other Bostan and Klinger (2018).

In German, a few dictionaries have been created for dimensional approaches. Among them is BAWL–R, a list of words rated with arousal, valence and imageability features Vo et al. (2009); Briesemeister et al. (2011), where the nouns of the lexicon have been assigned to emotion intensities, amongst other values. Still, German resources are rare in comparison to English ones. To our knowledge, corpora with sentence-wise emotion annotations are not available for this language.

In particular, there is no German corpus with speakers’ descriptions of emotionally intense events similar to the English ISEAR. ISEAR, the “International Survey on Emotion Antecedents and Reactions” (Scherer and Wallbott, 1997), was conducted by a group of psychologists who collected emotion data in the form of self-reports. The aim of the survey was to probe that emotions are invariant over cultures, and are characterized by patterns of bodily and behavioral changes (e.g., change in breathing, felt temperature, speech behaviors). In order to investigate such view, they administered an anonymous questionnaire to 3000 students all over the world, in which participants were asked to reconstruct an emotion episode associated to one of seven basic emotions (anger, disgust, fear, guilt, joy, sadness, shame), and to recall both their evaluation of the stimulus and their reaction to it. For the final dataset, all the reports were translated to English, and accordingly, the responses of, e.g., German speakers who took part in the survey are not available in their original language.

In this paper, we follow Scherer and Wallbott (1997) by re-using their set of seven basic emotions and recreating part of their questionnaire both in English and German. In contrast to ISEAR, we account for the fact that a description can be related to different emotions by its writer and its readers. Affective analyses have rendered evidence that emotional standpoints affect the quality of annotation tasks (Buechel and Hahn, 2017). For instance, annotation results vary depending on whether workers are asked if a text is associated with an emotion and if it evokes an emotion, with the first phrasing downplaying the reader’s perspective and inducing higher inter-annotator agreement (Mohammad and Turney, 2013). We take notice of these findings to design our annotation guidelines.

3 Crowdsourcing-based Corpus Creation

We developed a two-phase crowdsourcing experiment: one for generating descriptions, the other for rating the emotions of the descriptions. Phase 1 can be understood as sampling from $P(\text{description}|\text{emotion})$, obtaining likely descriptions for given emotions. Phase 2 estimates $P(\text{emotion}|\text{description})$, evaluating the association between a given description and all emotions. The participants’ intuitions gathered this way are interpretable as a measure for the interpersonal validity of the descriptions, and as a point of comparison for our classification results.

The two crowdourcing phases targeted both German and English. This enabled us to tease apart the effects of the change of setup and change of language compared to the original ISEAR collection.

Phase 1: Generation. We used the Figure-Eight (https://www.figure-eight.com) crowdsourcing platform. Following the ISEAR questionnaire, we presented annotators with one of the seven emotions in Scherer and Wallbott’s setup, and asked them to produce a textual description of an event in which they felt that emotion. The task of description generation was formulated as one of sentence completion (e.g., “Ich fühlte Freude, als/weil/…”, “I felt joy when/because …”), after observing that this strategy made the job easier for laypersons, without inducing any restriction on sentence structure (for details, see Suppl. Mat., Section A). Further, we asked annotators to specify their gender (male, female, other), the temporal distance of the event (i.e., whether the event took place days, weeks, months, or years before the time of text production), and the intensity and duration of the ensuing emotion (i.e., whether the experience was not very intense, moderately intense, intense and very intense, and whether it lasted a few minutes, one hour, multiple hours, or more than one day). To obtain an English equivalent to deISEAR, we crowd-sourced the same set of questions in English, creating a comparable English corpus (enISEAR). The generation task was published in two slices (Nov/Dec 2018 and Jan 2019). It was crucial for data quality to restrict the countries of origin (for German, DE/A; for English, UK/IR) – this prevented a substantial number of non-native participants who are proficient users of machine translation services from submitting answers. For each generated description, we paid 15 cents (see Suppl. Material, Section A for details).

Phase 2: Emotion Labeling. To verify to what extent the collected descriptions convey the emotions for which they were produced, we presented a new set of annotators with ten randomly sampled descriptions, omitting the emotion word (e.g., “I felt … when/because …”), together with the list of seven emotions. The task was to choose the emotion the original author most likely felt during the described event. Each description was judged by 5 annotators. We paid 15 cents per task.

4 Corpus Analysis

Descriptive analysis. We include all descriptions from Phase 1 in the final resource and the upcoming discussion, regardless of the inter-annotator agreement from Phase 2. Both deISEAR and enISEAR comprise 1001 event-centered descriptions: deISEAR includes 1084 sentences and 2613 distinct tokens, with a 0.19 type-token ratio; enISEAR contains 1366 sentences and a vocabulary of 3066 terms, with a type-token ratio of 0.12. Table 1 summarizes the Phase 1 annotation. For each prompting label111Transl. de$\rightarrow$en: Angst-Fear, Ekel-Disgust, Freude-Joy, Scham-Shame, Schuld-Guilt, Traurigkeit-Sadness, Wut-Anger, we report average description length, annotators’ gender, duration, intensity and temporal distance of the emotional events.

The main difference between the two languages is description length: English instances are almost twice as long (24.7 tokens) as German ones (13.2 tokens). These differences may be related to the differences in gender distribution between languages.

Most patterns are similar across German and English. In both corpora, Anger and Sadness receive the longest and shortest descriptions, respectively. Enraging facts are usually depicted through the specific aspects that irritated their experiencers, like “when a superior at work decided to make a huge issue out of something very petty just to […] prove they have power over me”. In contrast, sad events are reported with fewer details, possibly because they are often conventionally associated with pain and require little elaboration, such as “my grandmother had passed away”. Also the perceptual assessments of emotion episodes, as given by the extra-linguistic labels, are comparable between languages. The majority of descriptions are located at the high end of the scale both for intensity and temporal distance, i.e., they point to “milestone” events that are both remote and emotionally striking.

Agreement on emotions. We next analyze to what extent the emotions labelled in Phase 2 agree with the prompting emotion presented in Phase 1. Table 2 reports for how many descriptions (out of 143) the prompting emotion was selected one, two, three, four, or five (out of five) times in Phase 2. Agreement is similar between deISEAR and enISEAR. This indicates that the German items, although short, are sufficiently informative. In both languages, the agreement drops across the columns, yet half of the descriptions show perfect intersubjective validity (=5): 505 for German, 499 for English. We interpret this as a sign of quality.

Again, we find differences among emotions. Agreement is nearly perfect for Joy and rather low for Shame. These patterns can arise due to different processes. Certain emotions are easier to recognize from language (e.g., “when I saw someone else got stabbed near me”: Fear) than others (e.g. “when my daughter was rude to my wife”: elicited for Shame, arguably also associated with Anger or Sadness). Patterns may also indicate closer conceptual similarity among specific emotions (Russell and Mehrabian, 1977, cf.).

To follow up on this observation, Figure 1 shows two confusion matrices for German and English which plot the frequency with which annotators selected emotion labels (Phase 2, rows) for prompting emotions (Phase 1, columns). The results in the diagonals correspond to the =5 columns in Table 2, mirroring the overall high level of validity of the descriptions, and spanning the range between Joy (very high agreement) and Shame (low agreement). The off-diagonal cells indicate disagreements. In both languages, annotators perceive Shame descriptions as expressing Guilt, and vice versa (35% and 15% for English, 17% and 19% for German). In fact, Shame and Guilt “occur when events are attributed to internal causes” (Tracy and Robins, 2006), and thus they may appear overlapping.

We also see an interesting cross-lingual divergence. In deISEAR, Sadness is comparably often confused with Anger (13% of items), while in enISEAR it is Disgust that is regularly interpreted as Anger (25% of items). This might results from differences in the connotations of the prompting emotion words in the two languages. For Disgust (“Ekel”), German descriptions concentrate on physical repulsion, while the English descriptions also include metaphorical disgust which is more easily confounded with other emotions such as Anger.

Post-hoc Event type analysis. After the preceding analyses, we returned to the Phase 1 descriptions and performed a post-hoc annotation ourselves on a sample of 385 English and 385 German descriptions (balanced across emotions). We tagged them with dimensions motivated by Smith and Ellsworth (1985): whether the event was reoccurring (general), whether the event was in the future or in the past; whether it was a prospective emotion or actually felt; whether it had a social characteristic (involving other people or animals); whether the event had self consequences or consequences for others; and whether the author presumably had situational control or responsibility222One may be responsible, but not in control of the situation (e.g., “when I forgot to set an alarm”)..

Table 3 shows the results. In both English and German, only a few units depict general and future events, in line with the annotation guidelines. Fear more often targets the future than other emotions. Most event descriptions involve other participants, especially in English. In general, events seem to affect authors themselves more than other people, particularly in the case of Joy and Fear. Exceptions are Guilt and Sadness, for which there is a predominance of events whose effects bear down on others. Regarding the aspect of situational control, Shame and Guilt dominate. Guilt is particularly more frequent in descriptions in which the author is presumably responsible. These observations echo the findings by Tracy and Robins (2006).

Modeling. As a final analysis, we tested the compatibility of our created data with the original ISEAR corpus for emotion classification. We trained a maximum entropy classifier with L2 regularization with boolean unigram features on the original ISEAR corpus (7665 instances) and evaluated it on all instances collected in Phase 1 (with liblinear, Fan et al., 2008). We chose MaxEnt as a method as it constitutes are comparably strong baseline which is, in contrast to most neural classifiers, more easy to reproduce due to the convex optimization function and fewer hyper-parameters. We applied it to enISEAR and to a version of deISEAR translated with Google Translate333http://translate.google.com, applied on February 25, 2019, an effective baseline strategy for cross-lingual modeling (Barnes et al., 2016). In accord with the Phase 2 experiment, the emotion words present in the sentences were obscured. Table 4 shows a decent performance of the ISEAR model on our novel corpora, with similar scores and performance differences between emotion classes to previous studies (Bostan and Klinger, 2018).

Modeling performance and inter-annotator disagreement are correlated: emotions that are difficult to annotate are also difficult to predict (Spearman’s $\rho$ between $\text{F}_{1}$ and the diagonal in Figure 1 is 0.85 for German, $p=$ .01, and 0.75 for English, $p=$ .05). It is notable that results for German are on a level with English despite the translation step and the shorter length of the German descriptions. That goes against our expectations, as previous studies showed that translation is only sentiment-preserving to some degree Salameh et al. (2015); Lohar et al. (2018). We take this outcome as evidence for the cross-lingual comparability of deISEAR and enISEAR, and our general method.

5 Conclusion

We presented (a) deISEAR, a corpus of 1001 event descriptions in German, annotated with seven emotion classes; and (b) enISEAR, a companion English resource build analogously, to disentangle effects of annotation setup and English when comparing to the original ISEAR resource. Our two-phase annotation setup shows that perceived emotions can be different from expressed emotions in such event-focused corpus, which also affects classification performance.

Emotions vary substantially in their properties, both linguistic and extra-linguistic, which affects both annotation and modeling, while there is high consistency across the language pair English–German. Our modeling experiment shows that the straightforward application of machine translation for model transfer to another language does not lead to a drop in prediction performance.

Acknowledgments

This work was supported by Leibniz WissenschaftsCampus Tübingen “Cognitive Interfaces” and Deutsche Forschungsgemeinschaft (project SEAT, KL 2869/1-1). We thank Kai Sassenberg for inspiration and fruitful discussions.

Appendix A Corpus Generation and Labelling

For experimental reproducibility, we detail here our crowdsourcing approach. Figure 2 illustrates the instructions presented to the annotators for sentence generation (Phase 1), Figure 3 shows a preview of the task itself. The labelling task of Phase 2 is presented in Figure 4.

To built deISEAR, we targeted Figure-Eight contributors from Germany and Austria, while the English experiment was restricted to United Kingdom and Ireland. As a quality check, we required all workers to be level-3 contributors, i.e., the most experienced ones, who reached the highest accuracy in previous Figure-Eight jobs. It should be noted that these laypeople received only minimal and distant training, while participants of ISEAR were directly instructed by the experimenters. We aimed at adapting their questionnaire to a crowdsourcing framework, by formulating the task of sentence generation as one of sentence completion (e.g. “Ich fühlte Freude, als/weil/…”, “I felt Joy when/because …”). Preliminary experiments showed that people provided more coherent and grammatically correct sentences than when they were presented with a faithful translation of the original survey.

Phase 1 involved 121 English jobs and 116 German jobs after filtering unacceptable answers (e.g. nonsensical items), totalling 2002 tasks (hits). The two languages required a diverse amount of jobs because ungrammatical and nonsensical descriptions were (manually) discarded. In the second Phase, 34 jobs were launched for English and 23 for German. This way we collected 5005 annotations for each language (i.e. 5 annotations per description). Overall, data collection and annotation was finalized in three months. The total cost was 300$for Phase 1, and 150$ for Phase 2.

Appendix B Descriptive Analysis

Table 5 and Table 6 present a compact description of the corpora, normalizing the counts by column and by row blocks, as reported in Section 4 in the main paper.

Table 5 highlights differences in the distribution of emotions across different temporal distances, intensities, durations, and annotators’ gender. We see for instance that Shame is outstanding in English for long-distant events, while Anger and Disgust (depending on language) are more dominant in events that happened a few days prior to description production. For intensities, the distribution across emotions is most unbalanced for the label “Not Very”; for duration, Disgust is the prevailing emotion among those which lasted only a few minutes, while it is the less frequent among those which persisted for one or multiple days. The exact opposite holds for Joy and Sadness, which appear to be more durable states.

Table 6 highlights differences in the distribution of extra-linguistic labels across different emotions. A few commonalities emerge between the two languages. The majority of descriptions are referred to remote emotion episodes. Moreover, Anger-, Fear-, Joy- and Sadness-related descriptions are mostly about events which caused very intense affective states. For duration, most occurrences of Anger and Sadness lasted longer than one day both in German and English, while Fear episodes are more short-termed, similar to Disgust.

[TABLE]

[TABLE]

Appendix C Event-type Analysis

The event-type analysis presented in Section 4 targeted 385 items per language (55 descriptions per emotion). Table 2 in the paper shows the counts of instances associated to the psychological labels across the seven emotions.

For each description, we annotated the following boolean variables:

• •

About the event time:

• –

Does the text describe a general event?

• –

Does the text describe a future event?

• –

Does the text describe a past event?

• •

About the realization of the emotion:

• –

Is it an actual or a prospective emotion?

• •

About the embedding in a social environment:

• –

Are other people or animals part of the event description; is it a social event description?

• •

About the consequences of the event:

• –

Are there self-consequences?

• –

Are there consequences for others?

• •

About the control of the writer:

• –

Is the author presumably under situational control?

• –

Does the author presumably have self control/responsibility?

While the paper describes the distribution of labels by emotion, here we expand the discussion to the extra-linguistic information collected in Phase 1. Table 7 distributes the raw counts across the annotation values. It should be noticed that the random descriptions used for this analysis were not balanced with respect to their values of each variable. For this reason, Table 8 reports relative counts (i.e. counts of descriptions normalized by the number of instances within the label Day, Week, Month etc.).

Some regularities can be observed cross all columns of Table 8. For instance, events which involved a purposeful participation of their experiencer are a minority in both languages (Sit. control), and approximately 50% of the descriptions mention individuals other than the writer (Social). The latter proportion, however, is higher for English than for German.

Events that are linked to consequences for the self mostly come from the German sample (Self conseq.). In German, moreover, such type of events are recalled more frequently than events that had consequences on others (Conseq. oth.). The opposite is true for English: emotions of English authors often wrote about events that affected the life of other people or animals. This holds irrespective of the temporal distance, the intensity, the duration of the experience and the gender of the experiencer. Exceptions are English descriptions of facts which only lasted a few minutes, and which appear to bring consequences for the self more than for others (Self conseq. and Conseq. oth. in column min).

As for the responsibility of events, this label is consistent across all columns in the German sample. Instead, in English we observe some marked differences. Emotions with a low intensity (column NV) followed an event which was directly triggered by their experiencer, but very intense emotions are less frequently associated to responsibility (column VI). Lastly, shorter events (min) imply the responsibility dimension more than long ones ($\geq$d).

[TABLE]

[TABLE]

Appendix D Annotator Agreement

Section 4.1 discussed the agreement reached by different subsets of annotators at each generation label. We report relative counts in Table 9 and we extend the analysis in Table 10, summing over the prompting emotions. This table shows the interannotator agreement of Phase-2 annotators with respect to the meta-information given by the participants of Phase 1, i.e., all the alternatives for gender, intensity, duration and temporal distance under the column Labels.

These numbers represent the count of descriptions within a corpus – and not within a generation label, for which the annotation label is the same as the generation label. One can read the table as follows: 177 descriptions from deISEAR, which were labeled as VI by Phase 1 annotators, were then labelled by 5 Phase 2 annotators with their original prompting emotion; 506 instances provided by female annotators for enISEAR were labelled by at least 2 Phase 2 annotators with their original prompting emotion, and so on.

Notably, in the table of Section 4.2, the maximum value that each cell can reach is 143, i.e., the total number of descriptions prompted by a specific emotion. Here, the maximum value varies by cell, because each meta-data label is assigned to a different number of descriptions444For an overview of the distribution of meta-data labels over the descriptions, refer to Section 4.1.. Accordingly, higher counts do not necessarily indicate stronger agreement.

[TABLE]

[TABLE]

Appendix E Modeling

Table 11 shows the results of the maximum entropy classifier across all emotions.

[TABLE]