Motion-Conditioned Diffusion Model for Controllable Video Synthesis

TL;DR

MCDiff is a novel diffusion-based model that enables controllable video synthesis from a starting image and sparse motion inputs, achieving state-of-the-art quality and diversity.

Contribution

We introduce MCDiff, a conditional diffusion model that combines flow completion and diffusion techniques for controllable, stroke-guided video synthesis.

Findings

Achieves state-of-the-art visual quality in stroke-guided video synthesis.

Effectively predicts dense motion from sparse inputs using flow completion.

Demonstrates versatility on diverse content and motion in MPII Human Pose experiments.

Abstract

Recent advancements in diffusion models have greatly improved the quality and diversity of synthesized content. To harness the expressive power of diffusion models, researchers have explored various controllable mechanisms that allow users to intuitively guide the content synthesis process. Although the latest efforts have primarily focused on video synthesis, there has been a lack of effective methods for controlling and describing desired content and motion. In response to this gap, we introduce MCDiff, a conditional diffusion model that generates a video from a starting image frame and a set of strokes, which allow users to specify the intended content and dynamics for synthesis. To tackle the ambiguity of sparse motion inputs and achieve better synthesis quality, MCDiff first utilizes a flow completion model to predict the dense video motion based on the semantic understanding of the video frame and the sparse motion control. Then, the diffusion model synthesizes high-quality future frames to form the output video. We qualitatively and quantitatively show that MCDiff achieves the state-the-of-art visual quality in stroke-guided controllable video synthesis. Additional experiments on MPII Human Pose further exhibit the capability of our model on diverse content and motion synthesis.