On How Users Edit Computer-Generated Visual Stories

TL;DR

This study analyzes how human users edit AI-generated visual stories, revealing tendencies to shorten, diversify vocabulary, and replace nouns with pronouns, thereby informing future story generation system improvements.

Contribution

First comprehensive analysis of user edits on AI-generated stories, providing insights into editing behaviors to enhance story generation systems.

Findings

Users tend to shorten stories after editing.

Lexical diversity increases through editing.

Pronoun replacement is common in user edits.

Abstract

A significant body of research in Artificial Intelligence (AI) has focused on generating stories automatically, either based on prior story plots or input images. However, literature has little to say about how users would receive and use these stories. Given the quality of stories generated by modern AI algorithms, users will nearly inevitably have to edit these stories before putting them to real use. In this paper, we present the first analysis of how human users edit machine-generated stories. We obtained 962 short stories generated by one of the state-of-the-art visual storytelling models. For each story, we recruited five crowd workers from Amazon Mechanical Turk to edit it. Our analysis of these edits shows that, on average, users (i) slightly shortened machine-generated stories, (ii) increased lexical diversity in these stories, and (iii) often replaced nouns and their determiners/articles with pronouns. Our study provides a better understanding on how users receive and edit machine-generated stories,informing future researchers to create more usable and helpful story generation systems.

Tables

Table 1. Table 1. Average number of tokens with each POS tag per story. The universal part-of-speech tagset provided by NLTK toolkit was used. Δ Δ \Delta denoted the differences in average token numbers between between post- and pre-edit stories. The “DET” (determiner, article) tag contributes the most to the reduction of story length, and the “.” (punctuation marks) tag contributes the second.

	ADJ	ADP	ADV	CONJ	DET	NOUN	NUM	PRT	PRON	VERB	.	Total
Pre-Edit	3.09	3.46	1.92	0.50	8.11	10.00	0.02	1.62	2.14	7.00	5.16	43.02
Post-Edit	3.13	3.39	1.85	0.85	7.15	9.81	0.05	1.58	2.31	7.05	4.69	41.85
$\Delta$	0.04	-0.07	-0.07	0.35	-0.96	-0.20	0.03	-0.04	0.17	0.04	-0.47	-1.17

Full text

††journalyear: 2019††copyright: rightsretained††conference: CHI Conference on Human Factors in Computing Systems Extended Abstracts; May 4–9, 2019; Glasgow, Scotland UK††booktitle: CHI Conference on Human Factors in Computing Systems Extended Abstracts (CHI’19 Extended Abstracts), May 4–9, 2019, Glasgow, Scotland UK††doi: 10.1145/3290607.3312965††isbn: 978-1-4503-5971-9/19/05

1. Introduction

A large body of research in the AI community has been devoted to generating stories automatically. Many prior works aimed to generate stories based on text inputs, such as preceding sentences or writing prompts. For example, Roemmele et. al used a recurrent-neural-network architecture to generate stories in a sequence-to-sequence manner (Roemmele, 2016). Formulating a story as a sequence of events, Martin et. al proposed an event representation for neural-network-based story generation (Martin et al., 2018). Fan et al. created a hierarchical model that automatically generates stories conditioning on the writing prompts (Fan et al., 2018). Some other prior work explored generating stories based on images. For instance, Huang et. al introduced the visual storytelling task, in which the automatic model takes a sequence of photos as input, and generates a short story that narrates this photo sequence (Huang et al., 2016). An example visual story is shown in Figure 1. StyleNet (Gan et al., 2017) and SemStyle (Mathews et al., 2018) stylized descriptive image captions and made them more “attractive”, e.g., more romantic or humorous.

However, literature has little to say about how users would receive these machine-generated stories and put them to real use. More specifically, given the quality of stories created by modern automatic models, users will likely need to edit these stories for practical uses, for example, sharing them on social media. In this paper, we focus on the stories generated by modern visual storytelling models (Huang et al., 2016; Wang et al., 2018) and study how people would edit them. This study allows us to understand how text generation technologies can (or can not) help creative writing. Some prior work has explored interactive computer-supported story writing, in which the system populates suggestions or inspirations for the writer in near real-time when he/she writes the story. For example, the Creative Help system generates suggestions for the next sentence in the process of story writing (Swanson and Gordon, 2012; Roemmele and Gordon, 2015; Roemmele, 2016). Clark et al. studied machine-in-the-loop short story writing and concluded that machine intervention should balance between generating coherent and surprising suggestions (Clark et al., 2018). Our study provides a detailed understanding of how users receive and edit machine-generated text in the context of story writing.

2. Data Preparation

Machine-Generated Visual Stories

In the visual storytelling task, as introduced by Huang et al. (Huang et al., 2016), the computational model takes a sequence of five photos as input, and then automatically generates a short story describing the photo sequence. Huang et al. also released the VIST dataset111Visual Storytelling Dataset (VIST):

http://visionandlanguage.net/VIST/, which contains 81,743 unique Flickr photos in 20,211 sequences, aligned to human-written stories (as shown in Figure 1). Many researchers have proposed approaches to generate visual stories using this dataset. In this paper, we ran the visual storytelling model released by Wang et al. (Wang et al., 2018) on the test set of VIST to obtain machine-generated visual stories. Wang’s approach was one of the state-of-the-art methods that participated in the first Visual Storytelling Challenge (Mitchell et al., 2018) and also the earliest implementation that was made available online. Among the data in the test set of VIST, we removed the photo sequences containing any photos that have been deleted from Flickr by their owners and only used no more than one photo sequence per Flickr photo album to reduce redundancy. Eventually, we obtained 962 machine-generated visual stories, and each has a corresponding photo sequence.

Visual Story Post-Editing

For each story, we recruited five crowd workers from Amazon Mechanical Turk to edit it, respectively.222We specified the following qualifications workers must meet to work on our tasks: HIT (Human Intelligence Task) Approval Rate ¿= 98%, Number of HITs Approved ¿= 3000, and location = US. The Adult Content Qualification was also used. The following instruction was used to guide workers: “Please edit the story text as if these were your photos, and you would like using this story to share your experience with your friends.” We instructed workers to stick with the plot and the point of view (first-, second-, or third-person) of the original story so that workers will not abandon the machine-generated story and write a new one from scratch. The goal of this study is to understand how people edit machine-generated stories. Further research is required to explore factors, such as how coherent the story is, or which level of details does the story provide, that could affect user’ willingness to edit a story or abandon it. The worker interface is shown in Figure 3. The photo sequence was also displayed on the interface. The price of each task was $0.12. As a result, 197 workers generated 962$\times$ 5 = 4,810 edited stories. Figure 2 shows an example of a generated visual story, before and after human editing.

3. Analysis

We analyzed the length, type-token ratio (TTR), and the part of speech tags of pre- and post-edited visual stories. We used the NLTK toolkit to segment sentences, tokenize sentences into words (known as “tokens”), and tag part-of-speech (POS) for each word (Bird et al., 2009). It is noteworthy that the VIST dataset normalized the story text, e.g., replaced all female names such as “Amy” and “Sarah” with “[female]”. We replaced all “[female]” with “Amy” and “[male]” with “Tom” for our analysis. The detailed results and discussions are as follows.

3.1. Edited stories are slightly shorter.

On average, the edited visual stories are shorter significantly (paired t-test, $p<0.001$, $N=4810$). We calculated the number of tokens (also known as “sentence length”) of each story. An automatic story contains an average of 43.02 tokens (punctuation marks included, SD = 4.96), and an edited story contains an average of 41.85 tokens (SD = 9.70). This comparison indicates that users on average eliminated 1.17 tokens. Meanwhile, we noticed that the edited stories’ lengths resulted in a higher sample standard deviation (SD), which suggested that human-written stories are more diverse regarding story length.

3.2. Edited stories have a higher lexical diversity.

We also studied the lexical diversity of these storied. Type-token ratio (TTR) is the one of most commonly used index of lexical diversity (Hardie and McEnery, 2006), which is calculated as the ratio between the number of different words (types) $N_{type}$ and the total number of words (tokens) $N_{token}$ of a text unit, i.e., $TTR=N_{type}/N_{token}$. A higher TTR indicates a higher lexical diversity. Per our analysis, the average TTR for an automatic story is 0.62 (SD = 0.09), while the average TTR for an edited story is 0.72 (SD = 0.06). This difference is statistically significant (paired t-test, $p<0.001$, $N=4810$). This result suggested that users reduced the word redundancy and increased the lexical diversity in machine-generated stories. The normalized frequency histogram of TTR of pre- and post-edited stories are shown in Figure 4.

3.3. Nouns (NOUN) with determiners/articles (DET) are often replaced by pronouns (PRON).

We also analyzed the parts of speech of the words in each story. The universal part-of-speech tagset333A Universal Part-of-Speech Tagset:

https://www.nltk.org/book/ch05.html provided by NLTK toolkit was used. The average number of tokens of each POS in a story is shown in Table 1. We observed that the “DET” (determiner, article) tag contributes the most to the reduction of story length, while the number of “PRON” (pronoun) tags increased the most. In order to understand the possible reasons behind this phenomenon, we observed the stories that had fewer DET tags but more PRON tags after post editing. We found that the nouns with determiners and articles (DET) are often replaced by pronouns (PRON). The following is a typical example, where the replacement texts are highlighted in red.

[pre-edit] the car was parked in the middle of the road . …

[post-edit] we drove home and parked by lots of other cars . …

The following is another example:

[pre-edit] the group of friends went to the camp site to see what was going on . we had a lot of food to eat . the kids had a great time . we had a lot of fun . we had a lot of fun .

[post-edit] a group of friends went camping together . they took plenty of food to eat . they had a great time together . everyone had a lot of fun .

We also noticed that the “.” (punctuation marks) tags reduced significantly. This might be caused by the fact that machine-generated stories can be repetitive or too general, so users often merged two sentences or simply removed one of them. For instance, in the example above (the text highlighted in blue), the user decided to remove one sentence and thus one period mark (“.”) was eliminated.

4. Discussion & Future Work

Our analysis on the machine-generated and post-edited visual stories shows that, on average, users (i) slightly shortened machine-generated stories, (ii) increased lexical diversity, and (iii) often replaced nouns and their determiners/articles with pronouns, and merged or eliminated sentences. The higher-level theme emerged in these observations are text repetition in machine-generated stories. Generating redundant text is a known problem for many neural-network-based visual storytelling models, while some approaches (Hsu et al., 2018) suffer from this problem more than others (Kim et al., 2018). Our analysis indicates that, for the visual stories generated by Wang et al. (Wang et al., 2018), a significant part of human editing effort is to reduce word redundancy and increase lexical diversity. Furthermore, users frequently replaced nouns with pronouns suggests that the storytelling model should better understand which entities (e.g., human, building, animal) have been mentioned and thus can be called using pronouns (e.g., she, it). In the future, we will develop algorithms to learn from these edits to improve existing machine-generated stories. We will also study what types of machine-generated texts are more helpful in assisting users to compose short stories, aiming at building a human-centered computer-supported storytelling system.

5. Acknowledgements

We thank Jiawei Chen for her help. We also thank the workers on Mechanical Turk who participated in our experiments.