FreqPrior: Improving Video Diffusion Models with Frequency Filtering Gaussian Noise

TL;DR

FreqPrior introduces a frequency domain filtering technique for noise initialization in video diffusion models, significantly enhancing generation quality and efficiency by preserving motion and details.

Contribution

The paper presents FreqPrior, a novel frequency filtering approach for noise in diffusion models, improving video quality and reducing inference time.

Findings

Achieves highest quality and semantic scores on VBench

Reduces inference time with partial sampling

Enhances motion dynamics and detail preservation

Abstract

Text-driven video generation has advanced significantly due to developments in diffusion models. Beyond the training and sampling phases, recent studies have investigated noise priors of diffusion models, as improved noise priors yield better generation results. One recent approach employs the Fourier transform to manipulate noise, marking the initial exploration of frequency operations in this context. However, it often generates videos that lack motion dynamics and imaging details. In this work, we provide a comprehensive theoretical analysis of the variance decay issue present in existing methods, contributing to the loss of details and motion dynamics. Recognizing the critical impact of noise distribution on generation quality, we introduce FreqPrior, a novel noise initialization strategy that refines noise in the frequency domain. Our method features a novel filtering technique designed to address different frequency signals while maintaining the noise prior distribution that closely approximates a standard Gaussian distribution. Additionally, we propose a partial sampling process by perturbing the latent at an intermediate timestep during finding the noise prior, significantly reducing inference time without compromising quality. Extensive experiments on VBench demonstrate that our method achieves the highest scores in both quality and semantic assessments, resulting in the best overall total score. These results highlight the superiority of our proposed noise prior.