Accelerating Video Diffusion Models via Distribution Matching

TL;DR

This paper presents a new diffusion distillation framework that significantly reduces the number of sampling steps needed for high-quality video generation, making diffusion models more practical.

Contribution

It introduces a novel distribution matching and distillation method that enables efficient video diffusion with fewer steps, improving speed without sacrificing quality.

Findings

Achieves high-quality video generation in just four sampling steps.

Outperforms existing methods in efficiency and quality.

Uses a combination of video GAN loss and distribution matching.

Abstract

Generative models, particularly diffusion models, have made significant success in data synthesis across various modalities, including images, videos, and 3D assets. However, current diffusion models are computationally intensive, often requiring numerous sampling steps that limit their practical application, especially in video generation. This work introduces a novel framework for diffusion distillation and distribution matching that dramatically reduces the number of inference steps while maintaining-and potentially improving-generation quality. Our approach focuses on distilling pre-trained diffusion models into a more efficient few-step generator, specifically targeting video generation. By leveraging a combination of video GAN loss and a novel 2D score distribution matching loss, we demonstrate the potential to generate high-quality video frames with substantially fewer sampling steps. To be specific, the proposed method incorporates a denoising GAN discriminator to distil from the real data and a pre-trained image diffusion model to enhance the frame quality and the prompt-following capabilities. Experimental results using AnimateDiff as the teacher model showcase the method's effectiveness, achieving superior performance in just four sampling steps compared to existing techniques.