CoNo: Consistency Noise Injection for Tuning-free Long Video Diffusion

TL;DR

CoNo introduces a novel noise injection method with a look-back mechanism and long-term regularization to improve scene consistency in long video diffusion without retraining.

Contribution

It proposes the CoNo method, enhancing long video generation by modeling fine-grained scene transitions and maintaining content consistency without additional training.

Findings

Improves scene consistency in long video generation.

Effective under single- and multi-text prompts.

Reduces abrupt scene transitions.

Abstract

Tuning-free long video diffusion has been proposed to generate extended-duration videos with enriched content by reusing the knowledge from pre-trained short video diffusion model without retraining. However, most works overlook the fine-grained long-term video consistency modeling, resulting in limited scene consistency (i.e., unreasonable object or background transitions), especially with multiple text inputs. To mitigate this, we propose the Consistency Noise Injection, dubbed CoNo, which introduces the "look-back" mechanism to enhance the fine-grained scene transition between different video clips, and designs the long-term consistency regularization to eliminate the content shifts when extending video contents through noise prediction. In particular, the "look-back" mechanism breaks the noise scheduling process into three essential parts, where one internal noise prediction part is injected into two video-extending parts, intending to achieve a fine-grained transition between two video clips. The long-term consistency regularization focuses on explicitly minimizing the pixel-wise distance between the predicted noises of the extended video clip and the original one, thereby preventing abrupt scene transitions. Extensive experiments have shown the effectiveness of the above strategies by performing long-video generation under both single- and multi-text prompt conditions. The project has been available in https://wxrui182.github.io/CoNo.github.io/.