Edit-Your-Motion: Space-Time Diffusion Decoupling Learning for Video Motion Editing

TL;DR

This paper introduces Edit-Your-Motion, a novel video motion editing method that uses space-time diffusion decoupling and one-shot fine-tuning to improve motion fidelity and appearance consistency in unseen videos.

Contribution

It proposes a new approach combining DDIM inversion, a motion attention module, and a two-stage learning strategy to enhance motion editing in videos, especially for unseen cases.

Findings

Outperforms existing methods in qualitative and quantitative tests.

Reduces ghosting and distortion in in-the-wild video editing.

Enables effective motion editing with high fidelity and consistency.

Abstract

Existing diffusion-based methods have achieved impressive results in human motion editing. However, these methods often exhibit significant ghosting and body distortion in unseen in-the-wild cases. In this paper, we introduce Edit-Your-Motion, a video motion editing method that tackles these challenges through one-shot fine-tuning on unseen cases. Specifically, firstly, we utilized DDIM inversion to initialize the noise, preserving the appearance of the source video and designed a lightweight motion attention adapter module to enhance motion fidelity. DDIM inversion aims to obtain the implicit representations by estimating the prediction noise from the source video, which serves as a starting point for the sampling process, ensuring the appearance consistency between the source and edited videos. The Motion Attention Module (MA) enhances the model's motion editing ability by resolving the conflict between the skeleton features and the appearance features. Secondly, to effectively decouple motion and appearance of source video, we design a spatio-temporal two-stage learning strategy (STL). In the first stage, we focus on learning temporal features of human motion and propose recurrent causal attention (RCA) to ensure consistency between video frames. In the second stage, we shift focus on learning the appearance features of the source video. With Edit-Your-Motion, users can edit the motion of humans in the source video, creating more engaging and diverse content. Extensive qualitative and quantitative experiments, along with user preference studies, show that Edit-Your-Motion outperforms other methods.