AniSora: Exploring the Frontiers of Animation Video Generation in the Sora Era

TL;DR

AniSora introduces a comprehensive system for animation video generation, including data processing, controllable modeling, and an evaluation benchmark, addressing the unique challenges of animated content creation and assessment.

Contribution

The paper presents AniSora, a novel framework with a large high-quality dataset, a specialized spatiotemporal model, and a new benchmark for animation video generation.

Findings

Supports image-to-video generation, frame interpolation, and localized animation.

Includes a dataset of over 10 million high-quality animation images.

Provides a new evaluation benchmark with 948 animation videos.

Abstract

Animation has gained significant interest in the recent film and TV industry. Despite the success of advanced video generation models like Sora, Kling, and CogVideoX in generating natural videos, they lack the same effectiveness in handling animation videos. Evaluating animation video generation is also a great challenge due to its unique artist styles, violating the laws of physics and exaggerated motions. In this paper, we present a comprehensive system, AniSora, designed for animation video generation, which includes a data processing pipeline, a controllable generation model, and an evaluation benchmark. Supported by the data processing pipeline with over 10M high-quality data, the generation model incorporates a spatiotemporal mask module to facilitate key animation production functions such as image-to-video generation, frame interpolation, and localized image-guided animation. We also collect an evaluation benchmark of 948 various animation videos, with specifically developed metrics for animation video generation. Our entire project is publicly available on https://github.com/bilibili/Index-anisora/tree/main.