Harry Potter and the Action Prediction Challenge from Natural Language

TL;DR

This paper investigates predicting upcoming actions in textual scenes using Harry Potter stories, creating a new dataset and evaluating models like LSTM and logistic regression for action prediction accuracy.

Contribution

Introduces HPAC, a novel dataset for action prediction from text, and evaluates baseline models, highlighting their strengths and limitations.

Findings

LSTM performs best on frequent actions and large descriptions.

Logistic regression is effective on infrequent actions.

The dataset contains over 82,000 samples and 85 actions.

Abstract

We explore the challenge of action prediction from textual descriptions of scenes, a testbed to approximate whether text inference can be used to predict upcoming actions. As a case of study, we consider the world of the Harry Potter fantasy novels and inferring what spell will be cast next given a fragment of a story. Spells act as keywords that abstract actions (e.g. 'Alohomora' to open a door) and denote a response to the environment. This idea is used to automatically build HPAC, a corpus containing 82,836 samples and 85 actions. We then evaluate different baselines. Among the tested models, an LSTM-based approach obtains the best performance for frequent actions and large scene descriptions, but approaches such as logistic regression behave well on infrequent actions.

Tables

Table 1. Table 1: Examples from the Harry Potter books showing how spells map to reactions to the environment.

Text fragment	Action
Ducking under Peeves, they ran for their lives, right to the end of the corridor where they slammed into a door - and it was locked. ‘This is it!’ Ron moaned, as they pushed helplessly at the door, ‘We’re done for! This is the end!’ They could hear footsteps, Filch running as fast as he could toward Peeves’s shouts. ‘Oh, move over’, Hermione snarled. She grabbed Harry’s wand, tapped the lock, and whispered, ‘Alohomora’.	Unlock the door
And then, without warning, Harry’s scar exploded with pain. It was agony such as he had never felt in all his life; his wand slipped from his fingers as he put his hands over his face; his knees buckled; he was on the ground and he could see nothing at all; his head was about to split open. From far away, above his head, he heard a high, cold voice say, ‘Kill the spare.’ A swishing noise and a second voice, which screeched the words to the night: ‘Avada Kedavra’	Kill a target
Harry felt himself being pushed hither and thither by people whose faces he could not see. Then he heard Ron yell with pain. ‘What happened?’ said Hermione anxiously, stopping so abruptly that Harry walked into her. ‘Ron, where are you? Oh, this is stupid’ - ‘Lumos’	Turn on a light

Table 2. Table 2: Corpus statistics: s 𝑠 s is the length of the snippet.

Statistics	Training	Dev	Test
#Actions	85	83	84
#Samples	66 274	8 279	8 283
#Tokens (s=32)	2 111 180	263 573	263 937
#Unique tokens (s=32)	33 067	13 075	13 207
#Tokens (s=128)	8 329 531	1 040 705	1 041 027
#Unique tokens (s=128)	60 379	25 146	25 285

Table 3. Table 3: Macro and weighted F-scores over 5 runs.

Snippet	Model	Macro			Weighted
		P	R	F	P	R	F
-	Base	0.1	1.2	0.2	1.3	11.5	2.4
32	mlr	18.7	11.6	13.1	28.9	31.4	28.3
	mlp	19.1	9.8	10.3	31.7	32.1	28.0
	lstm	13.7	9.7	9.5	29.1	32.2	28.6
	cnn	9.9	7.8	7.3	24.6	29.2	24.7
64	mlr	20.6	12.3	13.9	29.9	32.1	29.0
	mlp	17.9	9.5	9.8	31.2	32.7	27.9
	lstm	13.3	10.3	10.2	30.3	33.9	30.4
	cnn	9.8	7.8	7.4	25.0	29.9	25.4
96	mlr	20.4	13.3	14.6	30.3	32.0	29.3
	mlp	16.9	9.5	9.8	30.2	32.6	27.8
	lstm	14.0	10.5	10.3	30.6	34.5	30.7
	cnn	10.2	7.1	6.9	25.2	29.4	24.4
128	mlr	19.6	12.1	12.9	30.0	31.7	28.2
	mlp	18.9	9.9	10.3	31.4	32.9	28.0
	lstm	14.4	10.5	10.5	31.3	35.1	31.1
	cnn	8.8	7.8	7.1	24.8	30.2	25.0

Table 4. Table 4: Averaged recall at k 𝑘 k over 5 runs.

Snippet	Model	R@1	R@2	R@5	R@10
-	Base	11.5	-	-	-
32	mlr	31.4	43.7	60.3	73.5
	mlp	32.1	44.3	61.5	74.9
	lstm	32.2	44.3	61.5	74.7
	cnn	29.2	41.1	58.1	71.6
64	mlr	32.1	44.9	61.9	74.3
	mlp	32.7	46.0	63.5	76.6
	lstm	33.9	46.1	63.1	75.7
	cnn	29.9	41.8	59.0	72.2
96	mlr	32.0	44.5	60.7	74.6
	mlp	32.6	45.6	63.4	76.6
	lstm	34.5	46.9	63.7	76.1
	cnn	29.3	41.9	59.5	72.8
128	mlr	31.7	44.5	61.0	74.3
	mlp	32.9	45.8	63.2	76.9
	lstm	35.1	47.4	64.4	76.9
	cnn	30.2	42.3	59.6	72.8

Table 5. Table 5: Performance on frequent (those that occur above the average) and infrequent actions.

Snippet	Model	Frequent			Infrequent
		Fwe	R@1	R@5	Fwe	R@1	R@5
	Base	3.7	14.5	-	0.0	0.0	-
32	mlr	35.8	37.1	70.5	14.8	9.5	23.0
	mlp	35.9	38.1	71.9	13.2	9.4	21.8
	lstm	37.1	38.4	71.6	11.7	8.6	23.0
	cnn	33.1	35.5	69.3	7.1	5.2	15.2
64	mlr	36.7	37.9	71.8	14.9	9.9	24.0
	mlp	36.4	39.2	74.5	11.0	7.9	21.6
	lstm	39.2	40.3	73.0	12.4	9.4	25.4
	cnn	33.9	36.4	70.6	6.9	5.2	15.1
96	mlr	36.4	37.4	70.1	17.1	11.7	25.1
	mlp	36.2	39.1	74.0	11.0	7.9	23.1
	lstm	39.6	41.1	73.7	12.4	9.6	25.8
	cnn	32.7	35.8	71.6	6.3	4.8	13.7
128	mlr	35.4	37.2	70.5	15.4	10.7	25.0
	mlp	36.5	39.5	74.0	11.1	8.2	22.3
	lstm	40.3	41.9	74.4	12.3	9.5	26.2
	cnn	33.7	36.9	71.4	6.5	5.0	14.6

Table 6. Table 6: Label distribution for the hpac corpus

Action	#Training	#Dev	#Test	Action	#Training	#Dev	#Test
avada kedavra	7937	986	954	crucio	7852	931	980
accio	4556	595	562	lumos	4159	505	531
stupefy	3636	471	457	obliviate	3200	388	397
expelliarmus	2998	377	376	legilimens	1938	237	247
expecto patronum	1796	212	242	protego	1640	196	229
sectumsempra	1596	200	189	alohomora	1365	172	174
incendio	1346	163	186	scourgify	1317	152	166
reducto	1313	171	163	imperio	1278	159	144
wingardium leviosa	1265	158	154	petrificus totalus	1253	175	134
silencio	1145	153	136	reparo	1124	159	137
muffliato	1005	108	92	aguamenti	796	84	86
finite incantatem	693	90	75	incarcerous	686	99	87
nox	673	82	80	riddikulus	655	81	88
diffindo	565	90	82	impedimenta	552	88	79
levicorpus	535	63	68	evanesco	484	53	59
sonorus	454	66	73	point me	422	57	69
episkey	410	55	59	confringo	359	52	48
engorgio	342	52	41	colloportus	269	26	41
rennervate	253	24	33	portus	238	22	31
tergeo	235	23	26	morsmordre	219	29	38
expulso	196	23	20	homenum revelio	188	30	24
mobilicorpus	176	20	14	relashio	174	20	27
locomotor	172	24	19	avis	166	17	29
rictusempra	159	16	26	impervius	149	26	13
oppugno	144	18	7	furnunculus	137	20	20
serpensortia	133	14	15	confundo	130	17	21
locomotor mortis	127	14	15	tarantallegra	126	11	17
reducio	117	13	22	quietus	108	15	17
langlock	99	12	19	geminio	78	5	10
ferula	78	6	10	orchideous	76	7	5
densaugeo	67	13	8	liberacorpus	63	7	5
aparecium	63	14	10	anapneo	62	6	5
flagrate	59	4	11	deletrius	59	12	6
obscuro	57	11	7	prior incantato	56	4	3
deprimo	51	2	2	specialis revelio	50	11	6
waddiwasi	45	5	8	protego totalum	44	9	5
duro	36	4	4	salvio hexia	36	8	5
defodio	34	2	6	piertotum locomotor	30	4	3
glisseo	26	4	3	mobiliarbus	25	3	4
repello muggletum	23	2	5	erecto	23	7	5
cave inimicum	19	5	2	descendo	19	0	1
protego horribilis	18	7	5	meteolojinx recanto	10	3	1
peskipiksi pesternomi	7	0	0

Full text

1 Introduction

Natural language processing (nlp) has achieved significant advances in reading comprehension tasks Chen et al. (2016); Salant and Berant (2017). These are partially due to embedding methods Mikolov et al. (2013); Devlin et al. (2018) and neural networks Rosenblatt (1958); Hochreiter and Schmidhuber (1997); Vaswani et al. (2017), but also to the availability of new resources and challenges. For instance, in cloze-form tasks Hermann et al. (2015); Bajgar et al. (2016), the goal is to predict the missing word given a short context. Weston et al. (2015) presented baBI, a set of proxy tasks for reading comprenhension. In the SQuAD corpus Rajpurkar et al. (2016), the aim is to answer questions given a Wikipedia passage. Kocisky et al. (2018) introduce NarrativeQA, where answering the questions requires to process entire stories. In a related line, Frermann et al. (2017) use fictional crime scene investigation data, from the CSI series, to define a task where the models try to answer the question: ‘who committed the crime?’.

In an alternative line of work, script induction Schank and Abelson (1977) has been also a useful approach to evaluate inference and semantic capabilities of nlp systems. Here, a model processes a document to infer new sequences that reflect events that are statistically probable (e.g. go to a restaurant, be seated, check the menu, …). For example, Chambers and Jurafsky (2008) introduce narrative event chains, a representation of structured knowledge of a set of events occurring around a protagonist. They then propose a method to learn statistical scripts, and also introduce two different evaluation strategies. With a related aim, Pichotta and Mooney (2014) propose a multi-event representation of statistical scripts to be able to consider multiple entities. These same authors Pichotta and Mooney (2016) have also studied the abilities of recurrent neural networks for learning scripts, generating upcoming events given a raw sequence of tokens, using bleu Papineni et al. (2002) for evaluation.

This paper explores instead a new task: action prediction from natural language descriptions of scenes. The challenge is addressed as follows: given a natural language input sequence describing the scene, such as a piece of a story coming from a transcript, the goal is to infer which action is most likely to happen next.

Contribution

We introduce a fictional-domain English corpus set in the world of Harry Potter novels. The domain is motivated by the existence of a variety of spells in these literary books, associated with keywords that can be seen as unambiguous markers for actions that potentially relate to the previous context. This is used to automatically create a natural language corpus coming from hundreds of users, with different styles, interests and writing skills. We then train a number of standard baselines to predict upcoming actions, a task that requires to be aware of the context. In particular, we test a number of generic models, from a simple logistic regression to neural models. Experiments shed some light about their strengths and weaknesses and how these are related to the frequency of each action, the existence of other semantically related actions and the length of the input story.

2 HPAC: The Harry Potter’s Action prediction Corpus

To build an action prediction corpus, we need to: (1) consider the set of actions, and (2) collect data where these occur. Data should come from different users, to approximate a real natural language task. Also, it needs to be annotated, determining that a piece of text ends up triggering an action. These tasks are however time consuming, as they require annotators to read vast amounts of large texts. In this context, machine comprehension resources usually establish a compromise between their complexity and the costs of building them Hermann et al. (2015); Kocisky et al. (2018).

2.1 Domain motivation

We rely on an intuitive idea that uses transcripts from the Harry Potter world to build up a corpus for textual action prediction. The domain has a set of desirable properties to evaluate reading comprehension systems, which we now review.

Harry Potter novels define a variety of spells. These are keywords cast by witches and wizards to achieve purposes, such as turning on a light (‘Lumos’), unlocking a door (‘Alohomora’) or killing (‘Avada Kedavra’). They abstract complex and non-ambiguous actions. Their use also makes it possible to build an automatic and self-annotated corpus for action prediction. The moment a spell occurs in a text represents a response to the environment, and hence, it can be used to label the preceding text fragment as a scene description that ends up triggering that action. Table 1 illustrates it with some examples from the original books.

This makes it possible to consider texts from the magic world of Harry Potter as the domain for the action prediction corpus, and the spells as the set of eligible actions.111Note that the corpus is built in an automatic way and some occurrences might not correspond to actions, but for example, to a description of the spell or even some false positive samples. Related to this, we have not censored the content of the stories, so some of them might contain adult content. Determining the length of the preceding context, namely snippet, that has to be considered as the scene description is however not trivial. This paper considers experiments (§4) using snippets with the 32, 64, 96 and 128 previous tokens to an action. We provide the needed scripts to rebuild the corpus using arbitrary lengths.222https://github.com/aghie/hpac

2.2 Data crawling

The number of occurrences of spells in the original Harry Potter books is small (432 occurrences), which makes it difficult to train and test a machine learning model. However, the amount of available fan fiction for this saga allows to create a large corpus. For hpac, we used fan fiction (and only fan fiction texts) from https://www.fanfiction.net/book/Harry-Potter/ and a version of the crawler by Milli and Bamman (2016).333Due to the website’s Terms of Service, the corpus cannot be directly released.

We collected Harry Potter stories written in English and marked with the status ‘completed’. From these we extracted a total of 82 836 spell occurrences, that we used to obtain the scene descriptions. Table 2 details the statistics of the corpus (see also Appendix A). Note that similar to Twitter corpora, fan fiction stories can be deleted over time by users or admins, causing losses in the dataset.444They also can be modified, making it unfeasible to retrieve some of the samples.

Preprocessing

We tokenized the samples with Manning et al. (2014) and merged the occurrences of multi-word spells into a single token.

3 Models

This work addresses the task as a classification problem, and in particular as a sequence to label classification problem. For this reason, we rely on standard models used for this type of task: multinomial logistic regression, a multi-layered perceptron, convolutional neural networks and long short-term memory networks. We outline the essentials of each of these models, but will treat them as black boxes. In a related line, Kaushik and Lipton (2018) discuss the need of providing rigorous baselines that help better understand the improvement coming from future and complex models, and also the need of not demanding architectural novelty when introducing new datasets.

Although not done in this work, an alternative (but also natural) way to address the task is as a special case of language modelling, where the output vocabulary is restricted to the size of the ‘action’ vocabulary. Also, note that the performance for this task is not expected to achieve a perfect accuracy, as there may be situations where more than one action is reasonable, and also because writers tell a story playing with elements such as surprise or uncertainty.

The source code for the models can be found in the GitHub repository mentioned above.

Notation

$w_{1:n}$ denotes a sequence of words $w_{1},...,w_{n}$ that represents the scene, with $w_{i}\in V$. $F_{\theta}(\cdot)$ is a function parametrized by $\theta$. The task is cast as $F:V^{n}\rightarrow A$, where $A$ is the set of actions.

3.1 Machine learning models

The input sentence $w_{1:n}$ is encoded as a one-hot vector, $\mathbf{v}$ (total occurrence weighting scheme).

Multinomial Logistic Regression

Let mlr${}_{\theta}(\mathbf{v})$ be an abstraction of a multinomial logistic regression parametrized by $\theta$, the output for an input $\mathbf{v}$ is computed as the $\operatorname*{arg\,max}_{a\in A}$ $P(y=a|\mathbf{v})$, where $P(y=a|\mathbf{v})$ is a $softmax$ function, i.e, $P(y=a|\mathbf{v})=\frac{e^{W_{a}\cdot\mathbf{v}}}{\sum_{a^{\prime}}^{A}e^{W_{a^{\prime}}\cdot\mathbf{v}}}$.

MultiLayer Perceptron

We use one hidden layer with a rectifier activation function ($relu(x)$=$max(0,x)$). The output is computed as mlp${}_{\theta}(\mathbf{v})$= $softmax(W_{2}\cdot relu(W\cdot\mathbf{v}+\mathbf{b})+\mathbf{b_{2}})$.

3.2 Sequential models

The input sequence is represented as a sequence of word embeddings, $\mathbf{w}_{1:n}$, where $\mathbf{w}_{i}$ is a concatenation of an internal embedding learned during the training process for the word $w_{i}$, and a pre-trained embedding extracted from GloVe Pennington et al. (2014)555http://nlp.stanford.edu/data/glove.6B.zip, that is further fine-tuned.

Long short-term memory network

Hochreiter and Schmidhuber (1997): The output for an element $\mathbf{w}_{i}$ also depends on the output of $\mathbf{w}_{i-1}$. The lstm${}_{\theta}(\mathbf{w}_{1:n})$666$n$ is set to be equal to the length of the snippet. takes as input a sequence of word embeddings and produces a sequence of hidden outputs, $\mathbf{h}_{1:n}$ ($\mathbf{h}_{i}$ size set to 128). The last output of the lstmθ, $\mathbf{h}_{n}$, is fed to a mlpθ.

Convolutional Neural Network

LeCun et al. (1995); Kim (2014). It captures local properties over continuous slices of text by applying a convolution layer made of different filters. We use a wide convolution, with a window slice size of length 3 and 250 different filters. The convolutional layer uses a $\mathit{relu}$ as the activation function. The output is fed to a max pooling layer, whose output vector is passed again as input to a mlpθ.

4 Experiments

Setup

All mlpθ’s have 128 input neurons and 1 hidden layer. We trained up to 15 epochs using mini-batches (size=16), Adam (lr=$0.001$) Kingma and Ba (2015) and early stopping.

Table 3 shows the macro and weighted F-scores for the models considering different snippet sizes.777As we have addressed the task as a classification problem, we will use precision, recall and F-score as the evaluation metrics. To diminish the impact of random seeds and local minima in neural networks, results are averaged across 5 runs.888Some macro F-scores do not lie within the Precision and Recall due to this issue. ‘Base’ is a majority-class model that maps everything to ‘Avada Kedavra’, the most common action in the training set. This helps test whether the models predict above chance performance. When using short snippets (size=32), disparate models such as our mlr, mlp and lstms achieve a similar performance. As the snippet size is increased, the lstm-based approach shows a clear improvement on the weighted scores999For each label, we compute their average, weighted by the number of true instances for each label. The F-score might be not between precision and recall., something that happens only marginally for the rest. However, from Table 3 it is hard to find out what the approaches are actually learning to predict.

To shed some light, Table 4 shows their performance according to a ranking metric, recall at $k$. The results show that the lstm-based approach is the top performing model, but the mlp obtains just slightly worse results. Recall at 1 is in both cases low, which suggests that the task is indeed complex and that using just lstms is not enough. It is also possible to observe that even if the models have difficulties to correctly predict the action as a first option, they develop certain sense of the scene and consider the right one among their top choices. Table 5 delves into this by splitting the performance of the model into infrequent and frequent actions (above the average, i.e. those that occur more than 98 times in the training set, a total of 20 actions). There is a clear gap between the performance on these two groups of actions, with a $\sim$50 points difference in recall at $5$. Also, a simple logistic regression performs similar to the lstm on the infrequent actions.

Error analysis101010Made over one of the runs from the lstm-based approach and setting the snippet size to 128 tokens.

Some of the misclassifications made by the lstm approach were semantically related actions and counter-actions. For example, ‘Colloportus’ (to close a door) was never predicted. The most common mis-classification (14 out of 41) was ‘Alohomora’ (to unlock a door), which was 5 times more frequent in the training corpus. Similarly, ‘Nox’ (to extinguish the light from a wand) was correctly predicted 6 times, meanwhile 36 mis-classifications correspond to ‘Lumos’ (to light a place using a wand), which was 6 times more frequent in the training set. Other less frequent spells that denote vision and guidance actions, such as ‘Point me’ (the wand acts a a compass pointing North) and ‘Homenum revelio’ (to revel a human presence) were also mainly misclassified as ‘Lumos’. This is an indicator that the lstm approach has difficulties to disambiguate among semantically related actions, especially if their occurrence was unbalanced in the training set. This issue is in line with the tendency observed for recall at $k$. Spells intended for much more specific purposes, according to the books, obtained a performance significantly higher than the average, e.g. F-score(‘Riddikulus’)=63.54, F-score(‘Expecto Patronum’)=55.49 and F-score(‘Obliviate’)=47.45. As said before, the model is significantly biased towards frequent actions. For 79 out of 84 gold actions in the test set, we found that the samples tagged with such actions were mainly classified into one of the top 20 most frequent actions.

Human comparison

We collected human annotations from 208 scenes involving frequent actions. The accuracy/F-macro/F-weighted was 39.20/30.00/40.90. The lstm approach obtained 41.26/25.37/39.86. Overall, the lstm approach obtained a similar performance, but the lower macro F-score by the lstm could be an indicator that humans can distinguish within a wider spectrum of actions. As a side note, super-human performance it is not strange in other nlp tasks, such as sentiment analysis Pang et al. (2002).

5 Conclusion

We explored action prediction from written stories. We first introduced a corpus set in the world of Harry Potter’s literature. Spells in these novels act as keywords that abstract actions. This idea was used to label a collection of fan fiction. We then evaluated standard nlp approaches, from logistic regression to sequential models such as lstms. The latter performed better in general, although vanilla models achieved a higher performance for actions that occurred a few times in the training set. An analysis over the output of the lstm approach also revealed difficulties to discriminate among semantically related actions.

The challenge here proposed corresponded to a fictional domain. A future line of work we are interested in is to test whether the knowledge learned with this dataset could be transferred to real-word actions (i.e. real-domain setups), or if such transfer is not possible and a model needs to be trained from scratch.

Acknowlegments

This work has received support from the TELEPARES-UDC project (FFI2014-51978-C2-2-R) and the ANSWER-ASAP project (TIN2017-85160-C2-1-R) from MINECO, and from Xunta de Galicia (ED431B 2017/01), and from the European Research Council (ERC), under the European Union’s Horizon 2020 research and innovation programme (FASTPARSE, grant agreement No 714150).

Appendix A Corpus distribution

Table 6 summarizes the label distribution across the training, development and test sets of the hpac corpus.