Infinite-Resolution Integral Noise Warping for Diffusion Models

TL;DR

This paper introduces an efficient infinite-resolution noise warping algorithm for diffusion models, improving temporal consistency in video generation while reducing computational costs significantly.

Contribution

It develops a novel algorithm based on Brownian bridges that achieves infinite-resolution accuracy with lower computational complexity compared to previous methods.

Findings

The new algorithm matches the accuracy of high-resolution methods.

It reduces computational costs by orders of magnitude.

Effective in real-world video generation applications.

Abstract

Adapting pretrained image-based diffusion models to generate temporally consistent videos has become an impactful generative modeling research direction. Training-free noise-space manipulation has proven to be an effective technique, where the challenge is to preserve the Gaussian white noise distribution while adding in temporal consistency. Recently, Chang et al. (2024) formulated this problem using an integral noise representation with distribution-preserving guarantees, and proposed an upsampling-based algorithm to compute it. However, while their mathematical formulation is advantageous, the algorithm incurs a high computational cost. Through analyzing the limiting-case behavior of their algorithm as the upsampling resolution goes to infinity, we develop an alternative algorithm that, by gathering increments of multiple Brownian bridges, achieves their infinite-resolution accuracy while simultaneously reducing the computational cost by orders of magnitude. We prove and experimentally validate our theoretical claims, and demonstrate our method's effectiveness in real-world applications. We further show that our method readily extends to the 3-dimensional space.