Training-free Motion Factorization for Compositional Video Generation

TL;DR

This paper introduces a training-free, motion factorization framework for compositional video generation that decomposes complex motion into three categories and guides synthesis accordingly.

Contribution

It proposes a novel, model-agnostic motion factorization approach that improves motion synthesis by explicitly modeling motion categories without additional training.

Findings

Achieves impressive motion synthesis performance on real-world benchmarks.

Effectively disentangles motion categories during video generation.

Framework is compatible with various diffusion models.

Abstract

Compositional video generation aims to synthesize multiple instances with diverse appearance and motion. However, current approaches mainly focus on binding semantics, neglecting to understand diverse motion categories specified in prompts. In this paper, we propose a motion factorization framework that decomposes complex motion into three primary categories: motionlessness, rigid motion, and non-rigid motion. Specifically, our framework follows a planning before generation paradigm. (1) During planning, we reason about motion laws on the motion graph to obtain frame-wise changes in the shape and position of each instance. This alleviates semantic ambiguities in the user prompt by organizing it into a structured representation of instances and their interactions. (2) During generation, we modulate the synthesis of distinct motion categories in a disentangled manner. Conditioned on the motion cues, guidance branches stabilize appearance in motionless regions, preserve rigid-body geometry, and regularize local non-rigid deformations. Crucially, our two modules are model-agnostic, which can be seamlessly incorporated into various diffusion model architectures. Extensive experiments demonstrate that our framework achieves impressive performance in motion synthesis on real-world benchmarks. Code is available at https://github.com/ZixuanWang0525/MF-CVG.