Follow-Your-Creation: Empowering 4D Creation through Video Inpainting

TL;DR

Follow-Your-Creation introduces a novel framework for 4D video creation and editing from monocular videos by reformulating the task as video inpainting, leveraging a foundation model, and ensuring temporal consistency and multi-view coherence.

Contribution

The paper presents a new 4D video creation method using a video inpainting foundation model, with a novel training strategy and inference modules for improved quality and editing capabilities.

Findings

Outperforms state-of-the-art in 4D video quality and versatility.

Enables prompt-based content editing with high consistency.

Effectively handles large camera motions during generation.

Abstract

We introduce Follow-Your-Creation, a novel 4D video creation framework capable of both generating and editing 4D content from a single monocular video input. By leveraging a powerful video inpainting foundation model as a generative prior, we reformulate 4D video creation as a video inpainting task, enabling the model to fill in missing content caused by camera trajectory changes or user edits. To facilitate this, we generate composite masked inpainting video data to effectively fine-tune the model for 4D video generation. Given an input video and its associated camera trajectory, we first perform depth-based point cloud rendering to obtain invisibility masks that indicate the regions that should be completed. Simultaneously, editing masks are introduced to specify user-defined modifications, and these are combined with the invisibility masks to create a composite masks dataset. During training, we randomly sample different types of masks to construct diverse and challenging inpainting scenarios, enhancing the model's generalization and robustness in various 4D editing and generation tasks. To handle temporal consistency under large camera motion, we design a self-iterative tuning strategy that gradually increases the viewing angles during training, where the model is used to generate the next-stage training data after each fine-tuning iteration. Moreover, we introduce a temporal packaging module during inference to enhance generation quality. Our method effectively leverages the prior knowledge of the base model without degrading its original performance, enabling the generation of 4D videos with consistent multi-view coherence. In addition, our approach supports prompt-based content editing, demonstrating strong flexibility and significantly outperforming state-of-the-art methods in both quality and versatility.