I2VControl: Disentangled and Unified Video Motion Synthesis Control

TL;DR

I2VControl introduces a unified framework for video motion synthesis that allows multiple control types to be combined without conflicts, using a point trajectory-based representation and adaptable architecture.

Contribution

The paper presents a novel disentangled, unified control framework that manages diverse video motion controls within a single pipeline, overcoming previous logical conflicts.

Findings

Achieves excellent performance across various control tasks.

Enables user-driven creative combinations in video synthesis.

Provides a plug-in adapter compatible with pre-trained models.

Abstract

Motion controllability is crucial in video synthesis. However, most previous methods are limited to single control types, and combining them often results in logical conflicts. In this paper, we propose a disentangled and unified framework, namely I2VControl, to overcome the logical conflicts. We rethink camera control, object dragging, and motion brush, reformulating all tasks into a consistent representation based on point trajectories, each managed by a dedicated formulation. Accordingly, we propose a spatial partitioning strategy, where each unit is assigned to a concomitant control category, enabling diverse control types to be dynamically orchestrated within a single synthesis pipeline without conflicts. Furthermore, we design an adapter structure that functions as a plug-in for pre-trained models and is agnostic to specific model architectures. We conduct extensive experiments, achieving excellent performance on various control tasks, and our method further facilitates user-driven creative combinations, enhancing innovation and creativity. Project page: https://wanquanf.github.io/I2VControl .