Zero-shot Generation of Coherent Storybook from Plain Text Story using Diffusion Models

TL;DR

This paper introduces a novel neural pipeline that uses large language models and diffusion models to generate coherent storybooks from plain text, emphasizing zero-shot capabilities and semantic editing for improved storytelling coherence.

Contribution

It presents a zero-shot storybook generation method combining language and diffusion models with semantic editing, avoiding expensive training on image-caption pairs.

Findings

Outperforms state-of-the-art image editing baselines

Ensures coherence in generated story sequences

Uses simple textual inversion techniques for zero-shot generation

Abstract

Recent advancements in large scale text-to-image models have opened new possibilities for guiding the creation of images through human-devised natural language. However, while prior literature has primarily focused on the generation of individual images, it is essential to consider the capability of these models to ensure coherency within a sequence of images to fulfill the demands of real-world applications such as storytelling. To address this, here we present a novel neural pipeline for generating a coherent storybook from the plain text of a story. Specifically, we leverage a combination of a pre-trained Large Language Model and a text-guided Latent Diffusion Model to generate coherent images. While previous story synthesis frameworks typically require a large-scale text-to-image model trained on expensive image-caption pairs to maintain the coherency, we employ simple textual inversion techniques along with detector-based semantic image editing which allows zero-shot generation of the coherent storybook. Experimental results show that our proposed method outperforms state-of-the-art image editing baselines.