Generating coherent comic with rich story using ChatGPT and Stable Diffusion

TL;DR

This paper presents a method combining ChatGPT and Stable Diffusion to generate coherent comics with rich stories and consistent art styles, introducing new evaluation metrics and achieving state-of-the-art results.

Contribution

The paper introduces a novel pipeline for comic generation using large language models and diffusion models, with new evaluation metrics and fine-tuning techniques for improved fidelity.

Findings

Achieved state-of-the-art character fidelity and art style consistency.

Developed a new evaluation method for AI-generated stories.

Successfully integrated story generation with image synthesis.

Abstract

Past work demonstrated that using neural networks, we can extend unfinished music pieces while maintaining the music style of the musician. With recent advancements in large language models and diffusion models, we are now capable of generating comics with an interesting storyline while maintaining the art style of the artist. In this paper, we used ChatGPT to generate storylines and dialogue and then generated the comic using stable diffusion. We introduced a novel way to evaluate AI-generated stories, and we achieved SOTA performance on character fidelity and art style by fine-tuning stable diffusion using LoRA, ControlNet, etc.