Reading Between the Lines: Exploring Infilling in Visual Narratives

TL;DR

This paper introduces a novel infilling approach for generating coherent visual narratives, supported by a large-scale dataset, resulting in improved storytelling quality over existing methods.

Contribution

It presents a new infilling technique for visual narrative generation and introduces the ViPT dataset with 46,200 procedures and 340k image-text pairs.

Findings

Achieved a METEOR score of 27.51, surpassing state-of-the-art in visual storytelling.

Demonstrated improved coherence in generated narratives using infilling.

Showcased the effectiveness of infilling in handling missing steps and images.

Abstract

Generating long form narratives such as stories and procedures from multiple modalities has been a long standing dream for artificial intelligence. In this regard, there is often crucial subtext that is derived from the surrounding contexts. The general seq2seq training methods render the models shorthanded while attempting to bridge the gap between these neighbouring contexts. In this paper, we tackle this problem by using \textit{infilling} techniques involving prediction of missing steps in a narrative while generating textual descriptions from a sequence of images. We also present a new large scale \textit{visual procedure telling} (ViPT) dataset with a total of 46,200 procedures and around 340k pairwise images and textual descriptions that is rich in such contextual dependencies. Generating steps using infilling technique demonstrates the effectiveness in visual procedures with more coherent texts. We conclusively show a METEOR score of 27.51 on procedures which is higher than the state-of-the-art on visual storytelling. We also demonstrate the effects of interposing new text with missing images during inference. The code and the dataset will be publicly available at https://visual-narratives.github.io/Visual-Narratives/.