Shape-for-Motion: Precise and Consistent Video Editing with 3D Proxy

TL;DR

Shape-for-Motion introduces a 3D proxy-based framework for precise, consistent, and controllable video editing, enabling complex manipulations across frames with high fidelity and physical consistency.

Contribution

The paper proposes a novel 3D proxy framework with a dual-propagation strategy for accurate, consistent video editing directly on 3D meshes, improving over existing methods.

Findings

Supports various editing operations like pose, rotation, scaling, and texture modifications.

Ensures high consistency and physical plausibility across video frames.

Demonstrates superior performance through extensive experiments.

Abstract

Recent advances in deep generative modeling have unlocked unprecedented opportunities for video synthesis. In real-world applications, however, users often seek tools to faithfully realize their creative editing intentions with precise and consistent control. Despite the progress achieved by existing methods, ensuring fine-grained alignment with user intentions remains an open and challenging problem. In this work, we present Shape-for-Motion, a novel framework that incorporates a 3D proxy for precise and consistent video editing. Shape-for-Motion achieves this by converting the target object in the input video to a time-consistent mesh, i.e., a 3D proxy, allowing edits to be performed directly on the proxy and then inferred back to the video frames. To simplify the editing process, we design a novel Dual-Propagation Strategy that allows users to perform edits on the 3D mesh of a single frame, and the edits are then automatically propagated to the 3D meshes of the other frames. The 3D meshes for different frames are further projected onto the 2D space to produce the edited geometry and texture renderings, which serve as inputs to a decoupled video diffusion model for generating edited results. Our framework supports various precise and physically-consistent manipulations across the video frames, including pose editing, rotation, scaling, translation, texture modification, and object composition. Our approach marks a key step toward high-quality, controllable video editing workflows. Extensive experiments demonstrate the superiority and effectiveness of our approach. Project page: https://shapeformotion.github.io/