Integrating Visuospatial, Linguistic and Commonsense Structure into Story Visualization

TL;DR

This paper enhances story visualization by integrating linguistic, commonsense, and visual structures into the generation process, leading to more coherent and relevant image sequences from narrative texts.

Contribution

It introduces a novel multi-structure encoding framework combining linguistic parse trees, commonsense knowledge, and visual cues, improving visual story generation quality.

Findings

Improved visual quality and consistency in generated stories.

Enhanced spatial and narrative relevance through structured inputs.

Significant metric improvements and positive human evaluations.

Abstract

While much research has been done in text-to-image synthesis, little work has been done to explore the usage of linguistic structure of the input text. Such information is even more important for story visualization since its inputs have an explicit narrative structure that needs to be translated into an image sequence (or visual story). Prior work in this domain has shown that there is ample room for improvement in the generated image sequence in terms of visual quality, consistency and relevance. In this paper, we first explore the use of constituency parse trees using a Transformer-based recurrent architecture for encoding structured input. Second, we augment the structured input with commonsense information and study the impact of this external knowledge on the generation of visual story. Third, we also incorporate visual structure via bounding boxes and dense captioning to provide feedback about the characters/objects in generated images within a dual learning setup. We show that off-the-shelf dense-captioning models trained on Visual Genome can improve the spatial structure of images from a different target domain without needing fine-tuning. We train the model end-to-end using intra-story contrastive loss (between words and image sub-regions) and show significant improvements in several metrics (and human evaluation) for multiple datasets. Finally, we provide an analysis of the linguistic and visuo-spatial information. Code and data: https://github.com/adymaharana/VLCStoryGan.