Pyramidal Flow Matching for Efficient Video Generative Modeling

TL;DR

This paper introduces a unified pyramidal flow matching algorithm for efficient high-resolution video generation, enabling end-to-end training and high-quality results with reduced computational resources.

Contribution

It proposes a novel pyramidal flow matching approach that unifies the generation process, improving efficiency and flexibility over previous cascaded architectures.

Findings

Supports high-quality 768p, 24 FPS video generation

Achieves 10-second videos within 20.7k GPU hours

Enables end-to-end training with a single Diffusion Transformer

Abstract

Video generation requires modeling a vast spatiotemporal space, which demands significant computational resources and data usage. To reduce the complexity, the prevailing approaches employ a cascaded architecture to avoid direct training with full resolution latent. Despite reducing computational demands, the separate optimization of each sub-stage hinders knowledge sharing and sacrifices flexibility. This work introduces a unified pyramidal flow matching algorithm. It reinterprets the original denoising trajectory as a series of pyramid stages, where only the final stage operates at the full resolution, thereby enabling more efficient video generative modeling. Through our sophisticated design, the flows of different pyramid stages can be interlinked to maintain continuity. Moreover, we craft autoregressive video generation with a temporal pyramid to compress the full-resolution history. The entire framework can be optimized in an end-to-end manner and with a single unified Diffusion Transformer (DiT). Extensive experiments demonstrate that our method supports generating high-quality 5-second (up to 10-second) videos at 768p resolution and 24 FPS within 20.7k A100 GPU training hours. All code and models are open-sourced at https://pyramid-flow.github.io.