MotionV2V: Editing Motion in a Video

TL;DR

MotionV2V introduces a novel method for editing existing videos by directly modifying sparse motion trajectories, enabling natural, flexible, and user-preferred video edits through a motion-conditioned diffusion model.

Contribution

The paper presents a new approach for video editing that uses sparse trajectory modifications and a motion-conditioned diffusion model, along with a dataset of motion counterfactuals for training.

Findings

Achieves over 65% user preference in comparisons

Enables edits starting at any timestamp with natural propagation

Demonstrates effective motion control in video editing

Abstract

While generative video models have achieved remarkable fidelity and consistency, applying these capabilities to video editing remains a complex challenge. Recent research has explored motion controllability as a means to enhance text-to-video generation or image animation; however, we identify precise motion control as a promising yet under-explored paradigm for editing existing videos. In this work, we propose modifying video motion by directly editing sparse trajectories extracted from the input. We term the deviation between input and output trajectories a "motion edit" and demonstrate that this representation, when coupled with a generative backbone, enables powerful video editing capabilities. To achieve this, we introduce a pipeline for generating "motion counterfactuals", video pairs that share identical content but distinct motion, and we fine-tune a motion-conditioned video diffusion architecture on this dataset. Our approach allows for edits that start at any timestamp and propagate naturally. In a four-way head-to-head user study, our model achieves over 65 percent preference against prior work. Please see our project page: https://ryanndagreat.github.io/MotionV2V