Minute-Long Videos with Dual Parallelisms

TL;DR

This paper introduces DualParal, a distributed inference method for diffusion transformer video models that parallelizes across both temporal frames and model layers, significantly reducing latency and memory costs for long video generation.

Contribution

We propose a novel distributed inference strategy that enables efficient, asynchronous, and scalable long video generation by parallelizing across frames and layers with synchronization and feature caching.

Findings

Achieves 6.54× lower latency on 8×RTX 4090 GPUs.

Reduces memory cost by 1.48× while maintaining quality.

Enables generation of 1025-frame videos with artifact-free quality.

Abstract

Diffusion Transformer (DiT)-based video diffusion models generate high-quality videos at scale but incur prohibitive processing latency and memory costs for long videos. To address this, we propose a novel distributed inference strategy, termed DualParal. The core idea is that, instead of generating an entire video on a single GPU, we parallelize both temporal frames and model layers across GPUs. However, a naive implementation of this division faces a key limitation: since diffusion models require synchronized noise levels across frames, this implementation leads to the serialization of original parallelisms. We leverage a block-wise denoising scheme to handle this. Namely, we process a sequence of frame blocks through the pipeline with progressively decreasing noise levels. Each GPU handles a specific block and layer subset while passing previous results to the next GPU, enabling asynchronous computation and communication. To further optimize performance, we incorporate two key enhancements. Firstly, a feature cache is implemented on each GPU to store and reuse features from the prior block as context, minimizing inter-GPU communication and redundant computation. Secondly, we employ a coordinated noise initialization strategy, ensuring globally consistent temporal dynamics by sharing initial noise patterns across GPUs without extra resource costs. Together, these enable fast, artifact-free, and infinitely long video generation. Applied to the latest diffusion transformer video generator, our method efficiently produces 1,025-frame videos with up to 6.54$\times$ lower latency and 1.48$\times$ lower memory cost on 8$\times$RTX 4090 GPUs.