StoryAgent: Customized Storytelling Video Generation via Multi-Agent Collaboration

TL;DR

StoryAgent is a multi-agent framework that improves customized storytelling video generation by ensuring subject consistency and integrating specialized agents for story design, storyboard creation, and video synthesis.

Contribution

It introduces a novel multi-agent system for CSVG, enhancing control and consistency, and proposes new methods like LoRA-BE for intra-shot temporal coherence.

Findings

Outperforms state-of-the-art methods in consistency metrics

Effectively maintains protagonist identity across shots

Enhances control over storytelling video synthesis

Abstract

The advent of AI-Generated Content (AIGC) has spurred research into automated video generation to streamline conventional processes. However, automating storytelling video production, particularly for customized narratives, remains challenging due to the complexity of maintaining subject consistency across shots. While existing approaches like Mora and AesopAgent integrate multiple agents for Story-to-Video (S2V) generation, they fall short in preserving protagonist consistency and supporting Customized Storytelling Video Generation (CSVG). To address these limitations, we propose StoryAgent, a multi-agent framework designed for CSVG. StoryAgent decomposes CSVG into distinct subtasks assigned to specialized agents, mirroring the professional production process. Notably, our framework includes agents for story design, storyboard generation, video creation, agent coordination, and result evaluation. Leveraging the strengths of different models, StoryAgent enhances control over the generation process, significantly improving character consistency. Specifically, we introduce a customized Image-to-Video (I2V) method, LoRA-BE, to enhance intra-shot temporal consistency, while a novel storyboard generation pipeline is proposed to maintain subject consistency across shots. Extensive experiments demonstrate the effectiveness of our approach in synthesizing highly consistent storytelling videos, outperforming state-of-the-art methods. Our contributions include the introduction of StoryAgent, a versatile framework for video generation tasks, and novel techniques for preserving protagonist consistency.