NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation

TL;DR

NUWA-XL introduces a parallel, diffusion-based architecture for extremely long video generation, enabling high-quality, coherent videos with significantly faster inference by training directly on long videos and employing a coarse-to-fine approach.

Contribution

The paper presents NUWA-XL, a novel diffusion over diffusion architecture that generates long videos in parallel, reducing inference time and training-inference gap compared to existing methods.

Findings

Achieves high-quality long videos with global and local coherence.

Reduces inference time from 7.55 minutes to 26 seconds for 1024 frames.

Introduces FlintstonesHD, a new benchmark for long video generation.

Abstract

In this paper, we propose NUWA-XL, a novel Diffusion over Diffusion architecture for eXtremely Long video generation. Most current work generates long videos segment by segment sequentially, which normally leads to the gap between training on short videos and inferring long videos, and the sequential generation is inefficient. Instead, our approach adopts a ``coarse-to-fine'' process, in which the video can be generated in parallel at the same granularity. A global diffusion model is applied to generate the keyframes across the entire time range, and then local diffusion models recursively fill in the content between nearby frames. This simple yet effective strategy allows us to directly train on long videos (3376 frames) to reduce the training-inference gap, and makes it possible to generate all segments in parallel. To evaluate our model, we build FlintstonesHD dataset, a new benchmark for long video generation. Experiments show that our model not only generates high-quality long videos with both global and local coherence, but also decreases the average inference time from 7.55min to 26s (by 94.26\%) at the same hardware setting when generating 1024 frames. The homepage link is \url{https://msra-nuwa.azurewebsites.net/}