Decoupled Video Generation with Chain of Training-free Diffusion Model Experts

TL;DR

ConFiner introduces a decoupled, efficient video generation framework using multiple diffusion experts, significantly reducing computational costs while producing high-quality, long, coherent videos.

Contribution

It proposes a novel decoupled approach with chain of training-free diffusion experts and coordinated denoising for efficient, high-quality video synthesis.

Findings

Surpasses Lavie and Modelscope in quality at 10% inference cost

Generates videos up to 600 frames long with coherence

Achieves high subjective and objective metrics

Abstract

Video generation models hold substantial potential in areas such as filmmaking. However, current video diffusion models need high computational costs and produce suboptimal results due to extreme complexity of video generation task. In this paper, we propose \textbf{ConFiner}, an efficient video generation framework that decouples video generation into easier subtasks: structure \textbf{con}trol and spatial-temporal re\textbf{fine}ment. It can generate high-quality videos with chain of off-the-shelf diffusion model experts, each expert responsible for a decoupled subtask. During the refinement, we introduce coordinated denoising, which can merge multiple diffusion experts' capabilities into a single sampling. Furthermore, we design ConFiner-Long framework, which can generate long coherent video with three constraint strategies on ConFiner. Experimental results indicate that with only 10\% of the inference cost, our ConFiner surpasses representative models like Lavie and Modelscope across all objective and subjective metrics. And ConFiner-Long can generate high-quality and coherent videos with up to 600 frames.