DiffSynth: Latent In-Iteration Deflickering for Realistic Video Synthesis

TL;DR

DiffSynth introduces a novel latent in-iteration deflickering framework and a patch blending algorithm to produce coherent, high-quality videos from diffusion models across various tasks, addressing flickering issues in video synthesis.

Contribution

The paper presents a new method that applies deflickering in the latent space and introduces a patch blending algorithm, improving video coherence in diffusion-based synthesis.

Findings

Effective reduction of flickering in generated videos

Applicable to diverse video synthesis tasks

High-quality results demonstrated in experiments

Abstract

In recent years, diffusion models have emerged as the most powerful approach in image synthesis. However, applying these models directly to video synthesis presents challenges, as it often leads to noticeable flickering contents. Although recently proposed zero-shot methods can alleviate flicker to some extent, we still struggle to generate coherent videos. In this paper, we propose DiffSynth, a novel approach that aims to convert image synthesis pipelines to video synthesis pipelines. DiffSynth consists of two key components: a latent in-iteration deflickering framework and a video deflickering algorithm. The latent in-iteration deflickering framework applies video deflickering to the latent space of diffusion models, effectively preventing flicker accumulation in intermediate steps. Additionally, we propose a video deflickering algorithm, named patch blending algorithm, that remaps objects in different frames and blends them together to enhance video consistency. One of the notable advantages of DiffSynth is its general applicability to various video synthesis tasks, including text-guided video stylization, fashion video synthesis, image-guided video stylization, video restoring, and 3D rendering. In the task of text-guided video stylization, we make it possible to synthesize high-quality videos without cherry-picking. The experimental results demonstrate the effectiveness of DiffSynth. All videos can be viewed on our project page. Source codes will also be released.