ViStoryBench: Comprehensive Benchmark Suite for Story Visualization

TL;DR

ViStoryBench is a comprehensive benchmark suite for evaluating story visualization models across diverse narratives, styles, and characters, addressing limitations of existing benchmarks.

Contribution

It introduces a richly annotated, multi-dimensional benchmark with automated metrics and human verification, enabling systematic evaluation of story visualization models.

Findings

Validated metrics through human studies.

Assessed a broad range of models systematically.

Highlighted strengths and weaknesses of current models.

Abstract

Story visualization aims to generate coherent image sequences that faithfully represent a narrative and match given character references. Despite progress in generative models, existing benchmarks remain narrow in scope, often limited to short prompts, lacking character references, or single-image cases, failing to reflect real-world narrative complexity and obscuring true model performance.We introduce ViStoryBench, a comprehensive benchmark designed to evaluate story visualization models across varied narrative structures, visual styles, and character settings. It features richly annotated multi-shot scripts derived from curated stories spanning literature, film, and folklore. Large language models assist in story summarization and script generation, with all outputs verified by humans for coherence and fidelity. Character references are carefully curated to maintain consistency across different artistic styles. ViStoryBench proposes a suite of multi-dimensional automated metrics to evaluate character consistency, style similarity, prompt alignment, aesthetic quality, and artifacts like copy-paste behavior. These metrics are validated through human studies and used to assess a broad range of open-source and commercial models, enabling systematic analysis and encouraging advances in visual storytelling.