DeCo: Decoupled Human-Centered Diffusion Video Editing with Motion Consistency

TL;DR

DeCo is a novel video editing framework that separately manipulates humans and backgrounds in videos, maintaining spatial-temporal consistency and addressing lighting issues, thus improving editing quality especially for long human-centered videos.

Contribution

DeCo introduces a decoupled approach for human and background editing using a parametric human prior and layered background, with enhanced geometry, texture, and lighting consistency.

Findings

Outperforms prior methods in qualitative and quantitative tests.

Maintains coherence in long human-centered videos.

Effectively handles lighting inconsistencies during editing.

Abstract

Diffusion models usher a new era of video editing, flexibly manipulating the video contents with text prompts. Despite the widespread application demand in editing human-centered videos, these models face significant challenges in handling complex objects like humans. In this paper, we introduce DeCo, a novel video editing framework specifically designed to treat humans and the background as separate editable targets, ensuring global spatial-temporal consistency by maintaining the coherence of each individual component. Specifically, we propose a decoupled dynamic human representation that utilizes a parametric human body prior to generate tailored humans while preserving the consistent motions as the original video. In addition, we consider the background as a layered atlas to apply text-guided image editing approaches on it. To further enhance the geometry and texture of humans during the optimization, we extend the calculation of score distillation sampling into normal space and image space. Moreover, we tackle inconsistent lighting between the edited targets by leveraging a lighting-aware video harmonizer, a problem previously overlooked in decompose-edit-combine approaches. Extensive qualitative and numerical experiments demonstrate that DeCo outperforms prior video editing methods in human-centered videos, especially in longer videos.