Fast and Memory-Efficient Video Diffusion Using Streamlined Inference

TL;DR

This paper introduces Streamlined Inference, a training-free framework that reduces memory and computation in video diffusion models, enabling high-quality video generation on standard hardware by exploiting temporal and spatial properties.

Contribution

The paper proposes a novel, training-free method combining Feature Slicer, Operator Grouping, and Step Rehash to significantly reduce memory and computation in video diffusion models.

Findings

Peak memory reduced from 42GB to 11GB on AnimateDiff.

Inference speed improved on consumer GPU (e.g., 2080Ti).

High-quality video generation achieved with lower resource requirements.

Abstract

The rapid progress in artificial intelligence-generated content (AIGC), especially with diffusion models, has significantly advanced development of high-quality video generation. However, current video diffusion models exhibit demanding computational requirements and high peak memory usage, especially for generating longer and higher-resolution videos. These limitations greatly hinder the practical application of video diffusion models on standard hardware platforms. To tackle this issue, we present a novel, training-free framework named Streamlined Inference, which leverages the temporal and spatial properties of video diffusion models. Our approach integrates three core components: Feature Slicer, Operator Grouping, and Step Rehash. Specifically, Feature Slicer effectively partitions input features into sub-features and Operator Grouping processes each sub-feature with a group of consecutive operators, resulting in significant memory reduction without sacrificing the quality or speed. Step Rehash further exploits the similarity between adjacent steps in diffusion, and accelerates inference through skipping unnecessary steps. Extensive experiments demonstrate that our approach significantly reduces peak memory and computational overhead, making it feasible to generate high-quality videos on a single consumer GPU (e.g., reducing peak memory of AnimateDiff from 42GB to 11GB, featuring faster inference on 2080Ti).