StoryWeaver: A Unified World Model for Knowledge-Enhanced Story Character Customization

TL;DR

StoryWeaver introduces a knowledge graph-based approach for consistent and semantically rich story visualization, effectively maintaining character identity and improving multi-character generation in AI-generated stories.

Contribution

The paper presents a novel Character Graph and knowledge-enhanced spatial guidance to improve character consistency and semantic accuracy in story visualization.

Findings

Achieves +9.03% DINO-I and +13.44% CLIP-T improvements.

Effectively maintains character identity across scenarios.

Outperforms existing methods in visual story coherence.

Abstract

Story visualization has gained increasing attention in artificial intelligence. However, existing methods still struggle with maintaining a balance between character identity preservation and text-semantics alignment, largely due to a lack of detailed semantic modeling of the story scene. To tackle this challenge, we propose a novel knowledge graph, namely Character Graph (\textbf{CG}), which comprehensively represents various story-related knowledge, including the characters, the attributes related to characters, and the relationship between characters. We then introduce StoryWeaver, an image generator that achieve Customization via Character Graph (\textbf{C-CG}), capable of consistent story visualization with rich text semantics. To further improve the multi-character generation performance, we incorporate knowledge-enhanced spatial guidance (\textbf{KE-SG}) into StoryWeaver to precisely inject character semantics into generation. To validate the effectiveness of our proposed method, extensive experiments are conducted using a new benchmark called TBC-Bench. The experiments confirm that our StoryWeaver excels not only in creating vivid visual story plots but also in accurately conveying character identities across various scenarios with considerable storage efficiency, \emph{e.g.}, achieving an average increase of +9.03\% DINO-I and +13.44\% CLIP-T. Furthermore, ablation experiments are conducted to verify the superiority of the proposed module. Codes and datasets are released at https://github.com/Aria-Zhangjl/StoryWeaver.