Noise Crystallization and Liquid Noise: Zero-shot Video Generation using Image Diffusion Models

TL;DR

This paper proposes a zero-shot method to generate consistent and controllable videos from image diffusion models by augmenting input noise, introducing techniques like noise crystallization and liquid noise, with applications in relighting and upscaling.

Contribution

It introduces novel zero-shot techniques for video generation using existing image diffusion models, enabling animation without additional training.

Findings

Noise crystallization improves temporal consistency.

Liquid noise offers greater flexibility at the cost of some consistency.

The methods enable applications like relighting and style transfer.

Abstract

Although powerful for image generation, consistent and controllable video is a longstanding problem for diffusion models. Video models require extensive training and computational resources, leading to high costs and large environmental impacts. Moreover, video models currently offer limited control of the output motion. This paper introduces a novel approach to video generation by augmenting image diffusion models to create sequential animation frames while maintaining fine detail. These techniques can be applied to existing image models without training any video parameters (zero-shot) by altering the input noise in a latent diffusion model. Two complementary methods are presented. Noise crystallization ensures consistency but is limited to large movements due to reduced latent embedding sizes. Liquid noise trades consistency for greater flexibility without resolution limitations. The core concepts also allow other applications such as relighting, seamless upscaling, and improved video style transfer. Furthermore, an exploration of the VAE embedding used for latent diffusion models is performed, resulting in interesting theoretical insights such as a method for human-interpretable latent spaces.