Generating Animations from Screenplays

TL;DR

This paper presents a novel text-to-animation system capable of handling complex sentences by integrating a sentence simplification step, resulting in more accurate and reasonable animations from screenplays, validated through quantitative metrics and user studies.

Contribution

Introduces a robust NLP pipeline with sentence simplification for generating animations from complex screenplay texts, improving over existing systems.

Findings

Sentence simplification outperforms existing systems in BLEU and SARI metrics.

68% of users find the generated animations reasonable.

System handles complex sentences better than prior approaches.

Abstract

Automatically generating animation from natural language text finds application in a number of areas e.g. movie script writing, instructional videos, and public safety. However, translating natural language text into animation is a challenging task. Existing text-to-animation systems can handle only very simple sentences, which limits their applications. In this paper, we develop a text-to-animation system which is capable of handling complex sentences. We achieve this by introducing a text simplification step into the process. Building on an existing animation generation system for screenwriting, we create a robust NLP pipeline to extract information from screenplays and map them to the system's knowledge base. We develop a set of linguistic transformation rules that simplify complex sentences. Information extracted from the simplified sentences is used to generate a rough storyboard and video depicting the text. Our sentence simplification module outperforms existing systems in terms of BLEU and SARI metrics.We further evaluated our system via a user study: 68 % participants believe that our system generates reasonable animation from input screenplays.

Tables

Table 1. Table 1: Linguistic rules for text simplification module

Syntactic Structure	Identify procedure	Transform procedure
Coordination	search if cc and conj in dependency tree	cut cc and conj link. If conj is verb, mark it as new root; else replace it with its sibling node.
Pre-Correlative Conjugation	locates position of keywords: “either”, “both”,“neither”	removed the located word from dependency tree
Appositive Clause	find appos token and its head (none)	glue appositive noun phrase with “to be”
Relative Clause	find relcl token and its head	cut appos link, then traverse from root. Then, if no “wh” word present, put head part after anchor part; else, we divide them into 5 subcases (Table 2)
Adverbial Clause Modifier	find advcl token and its head. Also conjuncts of head token	cut advcl edge. If advcl token does not have subject, add subject of root as advcl’s most-left child and remove prep and mark token. Then traverse from both root and advcl token
Inverted Clausal Subject	attr token has to be the child of head of csubj token	change position of actual verb and subject
Clausal Complement	find ccomp token in dependency tree	cut ccomp link, add subject to subordinate clause if necessary
Passive Voice	check presence of nsubjpass or csubjpass optionally for auxpass and agent	cut auxpass link if any. Cut nsubjpass or csubjpass link. Prepend subject token to verb token’s right children. Finally append suitable subject.
Open Clause Complement	find xcomp verb token adn actual verb token	if aux token presents, cut aux link, then replace xcomp-verb in subject’s children with actual-verb, traverse from actual-verb; else, cut xcomp link, traverse from xcomp-verb
Adjective Clause	find acl verb token and its head	cut acl link. Link subject node with it. Traverser from acl node

Table 2. Table 2: Syntactic simplification rules applied on example sentences.

Type	Example Input Sentence	System Output Sentence 1	System Output Sentence 2
Coordination	She LAUGHS, and[cc] gives[conj] Kevin a kiss.	She laughs.	She gives Kevin a kiss.
Pre-Correlative	It’s followed by another squad car, both[preconj] with sirens blaring.	It’s followed by another squad car, with sirens blaring.	–
Appositive	Kevin is reading a book the Bible[appos]	Kevin reads a book	The book is the Bible.
Relative-dobj	She pulls out a letter which[dobj] she hands[relcl] to Keven	Shee pulls out a letter	She hands a lettre to Kevin.
Relative-pobj	A reef encloses the cove where[pobj] he came[relcl] from.	A reef encloses the cove	he comes from the cove.
Relative-nsubj	Frank gestures to the SALESMAN, who[nsubj]’s waiting[relcl] on a woman	the SALESMAN waits on a woman.	Frank gestures to the SALESMAN.
Relative-advmod	Chuck is in the stage of exposure where[advmod] the personality splits[relcl]	Chuck is in the stage of exposure	the personality splits at exposure.
Relative-poss	The girl, whose[poss] name is[relcl] Helga, cowers.	The girl cowers	The girl ’s name is Helga
Relative-omit	Kim is the sexpot Peter saw[relcl] in Washington Square Park	Peter sees Kim in Washington Square Park.	Kim is the sexpot.
Adverbial	Jim panics as[advcl] his mom reacts, shocked.	Jim panics, shocked.	Jim’s mom reacts.
Adverbial-remove	Suddenly there’s a KNOCK at the door, immediately after[prep] which JIM’S MOM enters[advcl].	Suddenly there ’s a KNOCK at the door.	Immediately JIM ’S MOM enters.
Inverted Cl. Subject	Running[csubj] towards Oz is Steve Stifler	Steve Stifler runs towards Oz.	–
Clausal Component	The thing is, it actually sounds[ccomp] really good.	The thing is.(will be eliminated by the filter)	It actually sounds really good.
Passive Voice	They[nsubjpass] are suddenly illuminated by the glare of headlights.	Suddenly the glare of headlights illuminateds them.	–
Open Clausal	The sophomore comes running[xcomp] through the kitchen.	The sophomore runs through the kitchen.	The sophomore comes.
Adjective	Stifler has a toothbrush hanging[acl] from his mouth.	A toothbrush hangs from Stifler’s mouth.	Stifler has a toothbrush.

Table 3. Table 3: Differences between system output and annotator responses

Carl touches Ellie’s shoulder as the doctor explains. Ellie drops her head in her hands.
System output	Annotator \@slowromancapi@	Annotator \@slowromancapii@	$BLE{U}_2$( %)
Carl touches Ellie ’s shoulder	carl touches ellie’s shoulder	carl touches ellie’s shoulder.	38.73
the doctor explains	the doctor explains	the doctor is talking.	100
Ellie drops Ellie head in Ellie hands	ellie drops her head in her hands	ellie drops her head in her hands.	48.79

Table 4. Table 4: Results on syntactic simplification

	$BLEU$	$SARI$
NTS-w2v	61.45	36.04
YATS	58.83	48.75
Our System	67.68	50.65

Table 5. Table 5: Results on textual ARFs ( %)

Field $BLE{U}_1$ Field $BLE{U}_1$ owner 56.16 org_action 80.92 target 41.85 manner 84.84 prop 28.98 location 71.89 action 70.46 direction 70.83 emotion 57.89 Table 5: Results on textual ARFs ( %)

Field P R F1 s_time 86.49 68.63 76.53 rot. 82.04 81.16 81.60 duration 94.72 73.92 83.04 transl. 75.49 86.47 80.61 speed 94.41 79.50 86.32 Table 6: Result on Non-textual ARFs( %)

Table 6. Table 7: User Study Results

	Strongly Disagree	Disagree	Neutral	Agree	Strongly Agree
The animation shown in the video is a reasonable visualization of the text.	13.64  %	18.18  %	22.95  %	28.64  %	16.59  %
All the actions mentioned in the text are shown in the video.	15.68  %	20  %	22.96  %	20.68  %	20.68  %
All the actions shown in the video are mentioned in the text.	12.96  %	11.14  %	16.36  %	23.18  %	36.36  %

Table 7. Table 8: FSM Transition Rules

ID	Rule Summary
1	If input is uppercase and contains heading words such as ‘INT’, ‘EXT’. etc, return True, otherwise False
2	If input is uppercase and contains character words such as ‘CONT.’, ‘(O.S)’, or if #indentation $>$ most frequent #indentation, return True. Otherwise False
3	If input starts with $`{(}^{\prime }$ , return True, otherwise False
4	If input ends with $`){}^{\prime }$, return True, otherwise False
5	If , return True, otherwise False
6	If the input is uppercase and contains transition words such as ‘DISSOLVE’, ‘CUT TO’. etc, return True, otherwise False
7	If the input equals to ‘THE END’, return True. Otherwise False.

Table 8. Table 9: Example Model Outputs

COMPLEX: Another parent , Mike Munson , sits on the bench with a tablet and uses an app to track and analyze the team ’s shots.
NSELSTM-B: Another parent, Mike Munson, sits on the bench with a tablet and uses an app to track.
YATS: Another parent sits on the bench with a tablet and uses an app to track and examines the team’ s shots. This parent is Mike Munson.
OURS: Another parent is Mike Munson. Another parent sits on the bench with a tablet. Another parent uses an app.
COMPLEX: Stowell believes that even documents about Lincoln’s death will give people a better understanding of the man who was assassinated 150 years ago this April.
NSELSTM-B: Stowell believes that the discovery about Lincoln’s death will give people a better understanding of the man.
YATS: Stowell believes that even documents about Lincoln’ s death will give people a better reason of the man. This man was assassinated 150 years ago this April.
OURS: Stowell believes. Even documents about Lincoln ’s death give people a better understanding of the man. Somebody assassinates the man 150 years ago this April.

Equations

$$r_i^t=\underset{r_j^t}{\mathrm{argmin}}lev\_dist(s_i,r_j^t),\forall j\in \{1,\text{mathchar}46\text{mathchar}46\text{mathchar}46,m_a^t\}$$

Full text

1 Introduction

Generating animation from texts can be useful in many contexts e.g. movie script writing Ma and Kevitt (2006); Liu and Leung (2006); Hanser et al. (2010), instructional videos Lu and Zhang (2002), and public safety Johansson et al. (2004). Text-to-animation systems can be particularly valuable for screenwriting by enabling faster iteration, prototyping and proof of concept for content creators.

In this paper, we propose a text-to-animation generation system. Given an input text describing a certain activity, the system generates a rough animation of the text. We are addressing a practical setting, where we do not have any annotated data for training a supervised end-to-end system. The aim is not to generate a polished, final animation, but a pre-visualization of the input text. The purpose of the system is not to replace writers and artists, but to make their work more efficient and less tedious. We are aiming for a system which is robust and could be deployed in a production environment.

Existing text-to-animation systems for screenwriting (§2) visualize stories by using a pipeline of Natural Language Processing (NLP) techniques for extracting information from texts and mapping them to appropriate action units in the animation engine. The NLP modules in these systems translate the input text into predefined intermediate action representations and the animation generation engine produces simple animation from these representations.

Although these systems can generate animation from carefully handcrafted simple sentences, translating real screenplays into coherent animation still remains a challenge. This can be attributed to the limitations of the NLP modules used with regard to handling complex sentences. In this paper, we try to address the limitations of the current text-to-animation systems. Main contributions of this paper are:

• We propose a screenplay parsing architecture which generalizes well on different screenplay formats (§3.1).
• We develop a rich set of linguistic rules to reduce complex sentences into simpler ones to facilitate information extraction (§3.2).
• We develop a new NLP pipeline to generate animation from actual screenplays (§3).

The potential applications of our contributions are not restricted to just animating screenplays. The techniques we develop are fairly general and can be used in other applications as well e.g. information extraction tasks.

2 Related Work

Translating texts into animation is not a trivial task, given that neither the input sentences nor the output animations have a fixed structure. Prior work addresses this problem from different perspectives Hassani and Lee (2016).

CONFUCIUS Ma and Kevitt (2006) is a system that converts natural language to animation using the FDG parser Tapanainen and Järvinen (1997) and WordNet Miller (1995). ScriptViz Liu and Leung (2006) is another similar system, created for screenwriting. It uses the Apple Pie parser Sekine (1998) to parse input text and then recognizes objects via an object-specific reasoner. It is limited to sentences having conjunction between two verbs. SceneMaker Hanser et al. (2010) adopts the same NLP techniques as proposed in CONFUCIUS Ma and Kevitt (2006) followed by a context reasoning module. Similar to previously proposed systems, we also use dependency parsing followed by linguistic reduction (§3.2).

Recent advances in deep learning have pushed the state of the art results on different NLP tasks Honnibal and Johnson (2015); Wolf et al. (2018); He et al. (2017). We use pre-trained models for dependency parsing, coreference resolution and SRL to build a complete NLP pipeline to create intermediate action representations. For the action representation (§3.4), we use a key-value pair structure inspired by the PAR architecture Badler et al. (2000), which is a knowledge base of representations for actions performed by virtual agents.

Our work comes close to the work done in the area of Open Information Extraction (IE) Niklaus et al. (2018). In particular, to extract information, Clause-Based Open IE systems Del Corro and Gemulla (2013); Angeli et al. (2015); Schmidek and Barbosa (2014) reduce a complex sentence into simpler sentences using linguistic patterns. However, the techniques developed for these systems do not generalize well to screenplay texts, as these systems have been developed using well-formed and factual texts like Wikipedia, Penn TreeBank, etc. An initial investigation with the popular Open IE system OLLIE (Open Language Learning for Information Extraction) Mausam et al. (2012) did not yield good results on our corpus.

Previous work related to information extraction for narrative technologies includes the CARDINAL system Marti et al. (2018); Sanghrajka et al. (2018), as well as the conversational agent PICA Falk et al. (2018). They focus on querying knowledge from stories. The CARDINAL system also generates animations from input texts. However, neither of the tools can handle complex sentences. We build on the CARDINAL system. We develop a new NLP module to support complex sentences and leverage the animation engine of CARDINAL.

Recently, a number of end-to-end image generation systems have been proposed Mansimov et al. (2015); Reed et al. (2016). But these systems do not synthesize satisfactory images yet, and are not suitable for our application. It is hoped that the techniques proposed in this paper could be used for automatically generating labelled data (e.g. (text,video) pairs) for training end-to-end text-to-animation systems.

3 Text-to-Animation System

We adopt a modular approach for generating animations from screenplays. The general overview of our approach is presented in Figure 1. The system is divided into three modules:

• Script Parsing Module: Given an input screenplay text, this module automatically extracts the relevant text for generating the animation (§3.1).

• NLP Module: It processes the extracted text to get relevant information. This has two sub-modules:

• •

Text Simplification Module: It simplifies complex sentences using a set of linguistic rules (§3.2).

• •

Information Extraction Module: It extracts information from the simplified sentences into pre-defined action representations (§3.4).

• Animation Generation Module: It generates animation based on action representations (§3.5).

3.1 Script Parsing Module

Typically, screenplays or movie scripts or scripts (we use the terms interchangeably), are made of several scenes, each of which corresponds to a series of consecutive motion shots. Each scene contains several functional components111https://www.storysense.com/format.htm: Headings (time and location), Descriptions (scene description, character actions), Character Cues (character name before dialog), Dialogs (conversation content), Slug Lines (actions inserted into continuous dialog) and Transitions (camera movement). In many scripts, these components are easily identifiable by indentation, capitalization and keywords. We call these scripts well-formatted, and the remaining ones ill-formatted. We want to segment the screenplays into components and are mainly interested in the Descriptions component for animation generation.

Well-formatted Scripts: We initially tried ScreenPy Winer and Young (2017) to annotate the well-formatted scripts with the component labels. However, ScreenPy did not perform well on our data. We developed our own model, based on Finite State Machines (FSM), for parsing scripts (for details refer to Appendix A). Due to space limitations, we do not describe the FSM model; the key idea is that the model uses hand-crafted transition rules to segment the input screenplay and generates (paragraph, component name) pairs.

Ill-formatted Scripts: Taking all the (paragraph, component name) pairs generated by the FSM as ground truth, an SVM model is trained to segment ill-formatted screenplays with inconsistent indentations. For extracting features, each paragraph is encoded into a fix-sized vector using a pre-trained Universal Sentence Encoder. The SVM is trained and tested on a 9:1 train-test data split. The result shows an accuracy of 92.72 % on the test set, which is good for our purposes, as we are interested mainly in the Descriptions component.

Coreference Resolution: Screenplays contain a number of ambiguous entity mentions (e.g. pronouns). In order to link mentions of an entity, an accurate coreference resolution system is required. The extracted Descriptions components are processed with the NeuralCoref222github.com/huggingface/neuralcoref system. Given the text, it resolves mentions (typically pronouns) to the entity they refer to in the text. To facilitate entity resolution, we prepend each Description component with the Character Cues component which appears before it in the screenplay (e.g. [character]MARTHA: [dialog]“I knew it!” [description]She then jumps triumphantly $\rightarrow{}$ MARTHA. She then jumps triumphantly).

3.2 Text Simplification Module

In a typical text-to-animation system, one of the main tasks is to process the input text to extract the relevant information about actions (typically verbs) and participants (typically subject/object of the verb), which is subsequently used for generating animation. This works well for simple sentences having a single verb with one subject and one (optional) object. However, the sentences in a screenplay are complicated and sometimes informal. In this work, a sentence is said to be complicated if it deviates from easily extractable and simple subject-verb-object (and its permutations) syntactic structures and possibly has multiple actions mentioned within the same sentence with syntactic interactions between them. By syntactic structure we refer to the dependency graph of the sentence.

In the case of screenplays, the challenge is to process such complicated texts. We take the text simplification approach, i.e. the system first simplifies a complicated sentence and then extracts the relevant information. Simplification reduces a complicated sentence into multiple simpler sentences, each having a single action along with its participants, making it straightforward to extract necessary information.

Recently, end-to-end Neural Text Simplification (NTS) systems  Nisioi et al. (2017); Saggion (2017) have shown reasonable accuracy. However, these systems have been trained on factual data such as Wikipedia and do not generalize well to screenplay texts. Our experiments with such a pre-trained neural text simplification system did not yield good results (§5.1). Moreover, in the context of text-to-animation, there is no standard labeled corpus to train an end-to-end system.

There has been work on text simplification using linguistic rules-based approaches. For example, Siddharthan (2011) propose a set of rules to manipulate sentence structure to output simplified sentences using syntactic dependency parsing. Similarly, the YATS system Ferrés et al. (2016) implements a set of rules in the JAPE language Cunningham et al. (2000) to address six syntactic structures: Passive Constructions, Appositive Phrases, Relative Clauses, Coordinated Clauses, Correlated Clauses and Adverbial Clauses. Most of the rules focus on case and tense correction, with only 1-2 rules for sentence splitting. We take inspiration from the YATS system, and our system incorporates modules to identify and transform sentence structures into simpler ones using a broader set of rules.

In our system, each syntactic structure is handled by an Analyzer, which contains two processes: Identify and Transform. The Identify process takes in a sentence and determines if it contains a particular syntactic structure. Subsequently, the Transform process focuses on the first occurrence of the identified syntactic structure and then splits and assembles the sentence into simpler sentences. Both Identify and Transform use Part-of-Speech (POS) tagging and dependency parsing Honnibal and Montani (2017) modules implemented in spaCy 2.0333https://spacy.io

The simplification algorithm (Algorithm 1) starts with an input sentence and recursively processes it until no further simplification is possible. It uses a queue to manage intermediate simplified sentences, and runs in a loop until the queue is empty. For each sentence, the system applies each syntactic analyzer to Identify the corresponding syntactic structure in the sentence (line 14). If the result is positive, the sentence is processed by the Transform function to convert it to simple sentences (line 16). Each of the output sentences is pushed by the controller into the queue (line 19). The process is repeated with each of the Identify analyzers (line 13). If none of the analyzers can be applied, the sentence is assumed to be simple and it is pushed into the result list (line 21). We summarize linguistic rules in Table 1 and examples are given in Table 2. Next, we describe the coordination linguistic rules. For details regarding other rules, please refer to Appendix B.

Coordination: Coordination is used for entities having the same syntactic relation with the head and serving the same functional role (e.g. subj, obj, etc.). It is the most important component in our simplification system. The parser tags word units such as “and” and “as well as” with the dependency label cc, and the conjugated words as conj. Our system deals with coordination based on the dependency tag of the conjugated word.

In the case of coordination, the Identify function simply returns whether cc or conj is in the dependency graph of the input sentence. The Transform function manipulates the graph structure based on the dependency tags of the conjugated words as shown in Figure 2. If the conjugated word is a verb, then we mark it as another root of the sentence. Cutting cc and conj edges in the graph and traversing from this new root results in a new sentence parallel to the original one. In other cases, such as the conjugation between nouns, we simply replace the noun phrases with their siblings and traverse from root again.

3.3 Lexical Simplification

In order to generate animation, actions and participants extracted from simplified sentences are mapped to existing actions and objects in the animation engine. Due to practical reasons, it is not possible to create a library of animations for all possible actions in the world. We limit our library to a predefined list of 52 actions/animations, expanded to 92 by a dictionary of synonyms (§3.5). We also have a small library of pre-uploaded objects (such as “campfire”, “truck” and others).

To animate unseen actions not in our list, we use a word2vec-based similarity function to find the nearest action in the list. Moreover, we use WordNet Miller (1995) to exclude antonyms. This helps to map non-list actions (such as “squint at”) to the similar action in the list (e.g. “look”). If we fail to find a match, we check for a mapping while including the verb’s preposition or syntactic object. We also use WordNet to obtain hypernyms for further checks, when the similarity function fails to find a close-enough animation. Correspondingly, for objects, we use the same similarity function and WordNet’s holonyms.

Our list of actions and objects is not exhaustive. Currently, we do not cover actions which may not be visual. For out of list actions, we give the user a warning that the action cannot be animated. Nevertheless, this is a work in progress and we are working on including more animations for actions and objects in our knowledge base.

3.4 Action Representation Field (ARF): Information Extraction

For each of the simplified sentences, information is extracted and populated into a predefined key-value pair structure. We will refer to the keys of this structure as Action Representation Fields (ARFs). These are similar to entities and relations in Knowledge Bases. ARFs include: owner, target, prop, action, origin_action, manner, modifier_location, modifier_direction, start-time, duration, speed, translation, rotation, emotion, partial_start_time (for more details see Appendix C). This structure is inspired by the PAR Badler et al. (2000) architecture, but adapted to our needs.

To extract the ARFs from the simplified sentences, we use a Semantic Role Labelling (SRL) model in combination with some heuristics, for example creating word lists for duration, speed, translation, rotation, emotion. We use a pre-trained Semantic Role Labelling model444AllenNLP SRL model: https://github.com/allenai/allennlp based on a Bi-directional LSTM network He et al. (2017) with pre-trained ELMo embeddings Peters et al. (2018). We map information from each sentence to the knowledge base of animations and objects.

3.5 Animation Generation

We use the animation pipeline of the CARDINAL system. We plug in our NLP module in CARDINAL to generate animation. CARDINAL creates pre-visualizations of the text, both in storyboard form and animation. A storyboard is a series of pictures that demonstrates the sequence of scenes from a script. The animation is a 3-D animated video that approximately depicts the script. CARDINAL uses the Unreal game engine Games (2007) for generating pre-visualizations. It has a knowledge base of pre-baked animations (52 animations, plus a dictionary of synonyms, resulting in 92) and pre-uploaded objects (e.g. “campfire”, “tent”). It also has 3-D models which can be used to create characters.

4 Text-to-Animation Corpus

We initially used a corpus of Descriptions components from ScreenPy Winer and Young (2017), in order to study linguistic patterns in the movie script domain. Specifically, we used the “heading” and “transition” fields from ScreenPy’s published JSON output on 1068 movie scripts scraped from IMSDb. We also scraped screenplays from SimplyScripts and ScriptORama555http://www.simplyscripts.com and http://www.script-o-rama.com. After separating screenplays into well-formatted and ill-formatted, Descriptions components were extracted using our model (§3.1). This gave a corpus of Descriptions blocks from 996 screenplays.

The corpus contains a total of 525,708 Descriptions components. The Descriptions components contain a total of 1,402,864 sentences. Out of all the Descriptions components, 49.45 % (259,973) contain at least one verb which is in the animation list (henceforth called “action verbs”). Descriptions components having at least one action verb have in total 920,817 sentences. Out of these, 42.2 % (388,597) of the sentences contain action verbs. In the corpus, the average length of a sentence is 12 words.

5 Evaluation and Analysis

There are no standard corpora for text-to-animation generation. It is also not clear how should such systems be evaluated and what should be the most appropriate evaluation metric. Nevertheless, it is important to assess how our system is performing. We evaluate our system using two types of evaluation: Intrinsic and Extrinsic. Intrinsic evaluation is for evaluating the NLP pipeline of our system using the BLEU metric. Extrinsic evaluation is an end-to-end qualitative evaluation of our text-to-animation generation system, done via a user study.

5.1 Intrinsic Evaluation

To evaluate the performance of our proposed NLP pipeline, 500 Descriptions components from the test set were randomly selected. Three annotators manually translated these 500 Descriptions components into simplified sentences and extracted all the necessary ARFs from the simplified sentences. This is a time intensive process and took around two months. 30 % of the Descriptions blocks contain verbs not in the list of 92 animation verbs. There are approximately 1000 sentences in the test set, with average length of 12 words. Each Descriptions component is also annotated by the three annotators for the ARFs.

Taking inspiration from the text simplification community Nisioi et al. (2017); Saggion (2017), we use the BLEU score Papineni et al. (2002) for evaluating our simplification and information extraction modules. For each simplified sentence $s_{i}$ we have 3 corresponding references $r^{1}_{i},r^{2}_{i}$ and $r^{3}_{i}$. We also evaluate using the SARI Xu et al. (2016) score to evaluate our text simplification module.

5.1.1 Sentence Simplification

Each action block $a$ is reduced to a set of simple sentences $S_{a}=\{s_{1},s_{2},\mathchar 46\relax\mathchar 46\relax\mathchar 46\relax\mathchar 46\relax s_{n_{a}}\}$. And for the same action block $a$, each annotator $t$, $t\in\{1,2,3\}$ produces a set of simplified sentences $R^{t}_{a}=\{r^{t}_{1},r^{t}_{2},\mathchar 46\relax\mathchar 46\relax\mathchar 46\relax r^{t}_{m^{t}_{a}}\}$. Since the simplification rules in our system may not maintain the original ordering of verbs, we do not have sentence level alignment between elements in $S_{a}$ and $R^{t}_{a}$. For example, action block $a$ = He laughs after he jumps into the water is reduced by our system into two simplified sentences $S_{a}=\{s_{1}=\textit{He jumps into the water},s_{2}=\textit{He laughs}\}$ by the temporal heuristics, while $annotator_{3}$ gives us $R^{3}_{a}=\{r^{3}_{1}=\textit{He laughs},r^{3}_{2}=\textit{He jumps into the water}\}$. In such cases, sequentially matching $s_{i}$ to $r_{j}$ will result in a wrong (hypothesis, reference) alignment which is ($s_{1}$, $r^{3}_{1}$) and ($s_{2}$, $r^{3}_{2}$).

To address this problem, for each hypothesis $s_{i}\in S_{a}$, we take the corresponding reference $r^{t}_{i}\in R^{t}_{a}$ as the one with the least Levenshtein Distance Navarro (2001) to $s_{i}$, i.e,

[TABLE]

As per this alignment, in the above example, we will have correct alignments ($s_{1}$, $r^{3}_{2}$) and ($s_{2}$, $r^{3}_{1}$). Thus, for each simplified sentence $s_{i}$ we have 3 corresponding references $r^{1}_{i}$, $r^{2}_{i}$ and $r^{3}_{i}$. The aligned sentences are used to calculate corpus level BLEU score666We used NLTK’s API with default parameters: http://www.nltk.org/api/nltk.translate.html#nltk.translate.bleu_score.corpus_bleu and SARI score777Implementation available at https://github.com/cocoxu/simplification/.

The evaluation results for text simplification are summarized in Table 4. We compare against YATS Ferrés et al. (2016) and neural end-to-end text simplification system NTS-w2v Nisioi et al. (2017). YATS is also a rule-based text simplification system. As shown in Table 4, our system performs better than YATS on both the metrics, indicative of the limitations of the YATS system. A manual examination of the results also showed the same trend. However, the key point to note is that we are not aiming for text simplification in the conventional sense. Existing text simplification systems tend to summarize text and discard some of the information. Our aim is to break a complex sentence into simpler ones while preserving the information.

An example of a Descriptions component with $BLEU_{2}$ scores is given in Table 3. In the first simplified sentence, the space between Ellie and ’s causes the drop in the score. But it gives exactly the same answer as both annotators. In the second sentence, the system output is the same as the annotator @slowromancapi@’s answer, so the $BLEU_{2}$ score is 1. In the last case, the score is low, as annotators possibly failed to replace her with the actual Character Cue Ellie. Qualitative examination reveals, in general, that our system gives a reasonable result for the syntactic simplification module. As exemplified, BLEU is not the perfect metric to evaluate our system, and therefore in the future we plan to explore other metrics.

5.1.2 ARF Evaluation

We also evaluate the system’s output for action representation fields against gold annotations. In our case, some of the fields can have multiple (2 or 3) words such as owner, target, prop, action, origin_action, manner, location and direction. We use $BLEU_{1}$ as the evaluation metric to measure the BOW similarity between system output and ground truth references. The results are shown in Table 6.

In identifying owner, target and prop, the system tends to use a fixed long mention, while annotators prefer short mentions for the same character/object. The score of prop is relatively lower than all other fields, which is caused by a systematic SRL mapping error. The relatively high accuracy on the action field indicates the consistency between system output and annotator answers.

Annotation on the emotion ARF is rather subjective. Responses on the this field are biased and noisy. The $BLEU_{1}$ score on this is relatively low. For the other non-textual ARFs, we use precision and recall to measure the system’s behavior. Results are shown in Table 6. These fields are subjective: annotators tend to give different responses for the same input sentence.

rotation and translation have Boolean values. Annotators agree on these two fields in most of the sentences. The system, on the other hand, fails to identify actions involving rotation. For example, in the sentence “Carl closes CARL ’s door sharply” all four annotators think that this sentence involves rotation, which is not found by the system. This is due to the specificity of rules on identifying these two fields.

speed, duration and start_time have high precision and low recall. This indicates the inconsistency in annotators’ answers. For example, in the sentence “Woody runs around to the back of the pizza truck”, two annotators give 2 seconds and another gives 1 second in duration. These fields are subjective and need the opinion of the script author or the director. In the future, we plan to involve script editors in the evaluation process.

5.2 Extrinsic Evaluation

We conducted a user study to evaluate the performance of the system qualitatively. The focus of the study was to evaluate (from the end user’s perspective) the performance of the NLP component w.r.t. generating reasonable animations.

We developed a questionnaire consisting of 20 sentence-animation video pairs. The animations were generated by our system. The questionnaire was filled by 22 participants. On an average it took around 25 minutes for a user to complete the study.

We asked users to evaluate, on a five-point Likert scale Likert (1932), if the video shown was a reasonable animation for the text, how much of the text information was depicted in the video and how much of the information in the video was present in the text (Table 7). The 68.18 % of the participants rated the overall pre-visualization as neutral or above. The rating was 64.32 % (neutral or above) for the conservation of textual information in the video, which is reasonable, given limitations of the system that are not related to the NLP component. For the last question, 75.90 % (neutral or above) agreed that the video did not have extra information. In general, there seemed to be reasonable consensus in the responses. Besides the limitations of our system, disagreement can be attributed to the ambiguity and subjectivity of the task.

We also asked the participants to describe qualitatively what textual information, if any, was missing from the videos. Most of the missing information was due to limitations of the overall system rather than the NLP component: facial expression information was not depicted because the character 3-D models are deliberately designed without faces, so that animators can draw on them. Information was also missing in the videos if it referred to objects or actions that do not have a close enough match in the object list or animations list. Furthermore, the animation component only supports animations referring to a character or object as a whole, not parts, (e.g. “Ben raises his head” is not supported).

However, there were some cases where the NLP component can be improved. For example, lexical simplification failed to map the verb “watches” to the similar animation “look”. In one case, syntactic simplification created only two simplified sentences for a verb which had three subjects in the original sentence. In a few cases, lexical simplification successfully mapped to the most similar animation (e.g.“argue” to “talk”) but the participants were not satisfied - they were expecting a more exact animation. We plan to address these shortcomings in future work.

6 Conclusion and Future Work

In this paper, we proposed a new text-to-animation system. The system uses linguistic text simplification techniques to map screenplay text to animation. Evaluating such systems is a challenge. Nevertheless, intrinsic and extrinsic evaluations show reasonable performance of the system. The proposed system is not perfect, for example, the current system does not take into account the discourse information that links the actions implied in the text, as currently the system only processes sentences independently. In the future, we would like to leverage discourse information by considering the sequence of actions which are described in the text Modi and Titov (2014); Modi (2016). This would also help to resolve ambiguity in text with regard to actions Modi et al. (2017); Modi (2017). Moreover, our system can be used for generating training data which could be used for training an end-to-end neural system.

Acknowledgments

We would like to thank anonymous reviewers for their insightful comments. We would also like to thank Daniel Inversini, Isabel Simó Aynés, Carolina Ferrari, Roberto Comella and Max Grosse for their help and support in developing the Cardinal system. Mubbasir Kapadia has been funded in part by NSF IIS-1703883 and NSF S&AS-1723869.

Referencias

• Angeli et al. (2015)

Gabor Angeli, Melvin Jose Johnson Premkumar, and Christopher D Manning. 2015.

Leveraging linguistic structure for open domain information extraction.

In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), volume 1, pages 344–354.

• Badler et al. (2000)

Norman I. Badler, Rama Bindiganavale, Jan Allbeck, William Schuler, Liwei Zhao, and Martha Palmer. 2000.

Embodied conversational agents.

chapter Parameterized Action Representation for Virtual Human Agents, pages 256–284. MIT Press, Cambridge, MA, USA.

• Cunningham et al. (2000)

Hamish Cunningham, Diana Maynard, and Valentin Tablan. 2000.

Jape: a java annotation patterns engine.

• Del Corro and Gemulla (2013)

Luciano Del Corro and Rainer Gemulla. 2013.

Clausie: clause-based open information extraction.

In Proceedings of the 22nd international conference on World Wide Web, pages 355–366. ACM.

• Falk et al. (2018)

Jessica Falk, Steven Poulakos, Mubbasir Kapadia, and Robert W Sumner. 2018.

Pica: Proactive intelligent conversational agent for interactive narratives.

In Proceedings of the 18th International Conference on Intelligent Virtual Agents, pages 141–146. ACM.

• Ferrés et al. (2016)

Daniel Ferrés, Montserrat Marimon, Horacio Saggion, and Ahmed AbuRa’ed. 2016.

Yats: Yet another text simplifier.

In Natural Language Processing and Information Systems, pages 335–342, Cham. Springer International Publishing.

• Games (2007)

Epic Games. 2007.

Unreal engine.

Online: https://www. unrealengine. com.

• Hanser et al. (2010)

Eva Hanser, Paul Mc Kevitt, Tom Lunney, and Joan Condell. 2010.

Scenemaker: Intelligent multimodal visualization of natural language scripts.

In Proceedings of the 20th Irish Conference on Artificial Intelligence and Cognitive Science, AICS’09, pages 144–153, Berlin, Heidelberg. Springer-Verlag.

• Hassani and Lee (2016)

Kaveh Hassani and Won-Sook Lee. 2016.

Visualizing natural language descriptions: A survey.

ACM Comput. Surv., 49(1):17:1–17:34.

• He et al. (2017)

Luheng He, Kenton Lee, Mike Lewis, and Luke Zettlemoyer. 2017.

Deep semantic role labeling: What works and what’s next.

In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 473–483.

• Honnibal and Johnson (2015)

Matthew Honnibal and Mark Johnson. 2015.

An improved non-monotonic transition system for dependency parsing.

In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1373–1378, Lisbon, Portugal. Association for Computational Linguistics.

• Honnibal and Montani (2017)

Matthew Honnibal and Ines Montani. 2017.

spacy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing.

To appear.

• Johansson et al. (2004)

Richard Johansson, David Williams, Anders Berglund, and Pierre Nugues. 2004.

Carsim: a system to visualize written road accident reports as animated 3d scenes.

In Proceedings of the 2nd Workshop on Text Meaning and Interpretation, pages 57–64. Association for Computational Linguistics.

• Likert (1932)

R. Likert. 1932.

A technique for the measurement of attitudes.

Archives of Psychology, 22(140):1–55.

• Liu and Leung (2006)

Zhi-Qiang Liu and Ka-Ming Leung. 2006.

Script visualization (scriptviz): a smart system that makes writing fun.

Soft Computing, 10(1):34–40.

• Lu and Zhang (2002)

Ruqian Lu and Songmao Zhang. 2002.

Automatic generation of computeranimation: using AI for movie animation.

Springer-Verlag.

• Ma and Kevitt (2006)

Minhua Ma and Paul Kevitt. 2006.

Virtual human animation in natural language visualisation.

Artif. Intell. Rev., 25(1-2):37–53.

• Mansimov et al. (2015)

Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. 2015.

Generating images from captions with attention.

arXiv preprint arXiv:1511.02793.

• Marti et al. (2018)

Marcel Marti, Jodok Vieli, Wojciech Witoń, Rushit Sanghrajka, Daniel Inversini, Diana Wotruba, Isabel Simo, Sasha Schriber, Mubbasir Kapadia, and Markus Gross. 2018.

Cardinal: Computer assisted authoring of movie scripts.

In 23rd International Conference on Intelligent User Interfaces, pages 509–519. ACM.

• Mausam et al. (2012)

Mausam, Michael Schmitz, Robert Bart, Stephen Soderland, and Oren Etzioni. 2012.

Open language learning for information extraction.

In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 523–534. Association for Computational Linguistics.

• Miller (1995)

George A. Miller. 1995.

Wordnet: A lexical database for english.

COMMUNICATIONS OF THE ACM, 38:39–41.

• Modi (2016)

Ashutosh Modi. 2016.

Event embeddings for semantic script modeling.

In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning.

• Modi (2017)

Ashutosh Modi. 2017.

Modeling common sense knowledge via scripts.

Ph.D. thesis, Universität des Saarlandes.

• Modi and Titov (2014)

Ashutosh Modi and Ivan Titov. 2014.

Inducing neural models of script knowledge.

In Conference on Computational Natural Language Learning (CoNLL).

• Modi et al. (2017)

Ashutosh Modi, Ivan Titov, Vera Demberg, Asad Sayeed, and Manfred Pinkal. 2017.

Modelling semantic expectation: Using script knowledge for referent prediction.

Transactions of the Association for Computational Linguistics, 5:31–44.

• Navarro (2001)

Gonzalo Navarro. 2001.

A guided tour to approximate string matching.

ACM computing surveys (CSUR), 33(1):31–88.

• Niklaus et al. (2018)

Christina Niklaus, Matthias Cetto, André Freitas, and Siegfried Handschuh. 2018.

A survey on open information extraction.

arXiv preprint arXiv:1806.05599.

• Nisioi et al. (2017)

Sergiu Nisioi, Sanja Štajner, Simone Paolo Ponzetto, and Liviu P Dinu. 2017.

Exploring neural text simplification models.

In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 85–91.

• Papineni et al. (2002)

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.

Bleu: A method for automatic evaluation of machine translation.

In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, pages 311–318, Stroudsburg, PA, USA. Association for Computational Linguistics.

• Peters et al. (2018)

Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.

Deep contextualized word representations.

arXiv preprint arXiv:1802.05365.

• Reed et al. (2016)

Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. 2016.

Generative adversarial text to image synthesis.

arXiv preprint arXiv:1605.05396.

• Saggion (2017)

Horacio Saggion. 2017.

Automatic text simplification.

Synthesis Lectures on Human Language Technologies, 10(1):1–137.

• Sanghrajka et al. (2018)

Rushit Sanghrajka, Wojciech Witon, Sasha Schriber, Markus H Gross, and Mubbasir Kapadia. 2018.

Computer-assisted authoring for natural language story scripts.

In AAAI.

• Schmidek and Barbosa (2014)

Jordan Schmidek and Denilson Barbosa. 2014.

Improving open relation extraction via sentence re-structuring.

In LREC, pages 3720–3723.

• Sekine (1998)

Satoshi Sekine. 1998.

Corpus-based Parsing and Sublanguage Studies.

Ph.D. thesis, New York, NY, USA.

AAI9831764.

• Siddharthan (2011)

Advaith Siddharthan. 2011.

Text simplification using typed dependencies: A comparison of the robustness of different generation strategies.

In Proceedings of the 13th European Workshop on Natural Language Generation, ENLG ’11, pages 2–11, Stroudsburg, PA, USA. Association for Computational Linguistics.

• Tapanainen and Järvinen (1997)

Pasi Tapanainen and Timo Järvinen. 1997.

A non-projective dependency parser.

In Proceedings of the Fifth Conference on Applied Natural Language Processing, ANLC ’97, pages 64–71, Stroudsburg, PA, USA. Association for Computational Linguistics.

• Winer and Young (2017)

David Winer and R. Young. 2017.

Automated screenplay annotation for extracting storytelling knowledge.

• Wolf et al. (2018)

Thomas Wolf, James Ravenscroft, Julien Chaumond, and Maxwell Rebo. 2018.

Neuralcoref: Coreference resolution in spacy with neural networks.

Available online at https://github.com/huggingface/neuralcoref.

• Xu et al. (2016)

Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016.

Optimizing statistical machine translation for text simplification.

Transactions of the Association for Computational Linguistics, 4:401–415.

Appendix A

Appendix B: Algorithms

Appendix C: Action Representation Fields

Action Representation Fields (ARFs) in the demo sentence James gently throws a red ball to Alice in the restaurant from back, extracted with SRL:

• owner: James
• target: a red ball
• prop: to Alice
• action: throw
• origin_action: throws
• manner: gently
• modifier_location: in the restaurant
• modifier_direction: from back

In this case, our output for the prop and target is not correct; they should be swapped. This is one example where this module can introduce errors.

Additional ARFs, extracted heuristically:

• startTime: Calculated by current scene time
• duration: We have a pre-defined list of words that when appearing in the sentence, they will indicate a short duration (e.g “run” or “fast”) and therefore the duration will be set to 1 second; in contrast, for words like “slowly” we assign a duration of 4 seconds; otherwise, the duration is 2 seconds.
• speed: Similarly to duration, we have pre-defined lists of words that would affect the speed of the pre-baked animation: “angrily” would result in faster movement, but “carefully” in slower movement. We have 3 scales: 0.5, 1, 2 which corresponds to slow, normal and fast.
• translation: We have a list of actions which would entail a movement in from one place to another, e.g. “go”. If the value of the action exists in this list, it is set to True, otherwise False.
• rotation: If the action exists in our list of verbs that entail rotation, this field is True, otherwise False. Rotation refers to movement in place e.g. “turn” or “sit”.
• emotion: We find the closet neighbor of each word in the sentence in list of words that indicate emotion, using word vector similarity. If the similarity exceeds an empirically tested threshold, then we take the corresponding emotion word as the emotion field of this action.
• partial_start_time: an important field, since it controls the sequence order of each action. It determines which actions happen in parallel and which happen sequentially. This is still an open question. We solve this problem when doing sentence simplification. Together with the input sentence, current time is also fed into each Analyzer. There are several rules in some of the Analyzers to obtain temporal information. For example, in Line 5 of the Adverbial Clause Analyzer (c.f.3), we assign different temporal sequences for simplified actions. The algorithm is shown in Algorithm 15. The sign together with specific prepositions determines the change direction of current temporal id. In the Coordination Analyzer, the current temporal id changes when it encounters two verbs sharing same subject. Then the later action will get a bigger temporal id.