Follow-Your-Creation: Empowering 4D Creation through Video Inpainting
Yue Ma, Kunyu Feng, Xinhua Zhang, Hongyu Liu, David Junhao Zhang, Jinbo Xing, Yinhan Zhang, Ayden Yang, Zeyu Wang, Qifeng Chen

TL;DR
Follow-Your-Creation introduces a novel framework for 4D video creation and editing from monocular videos by reformulating the task as video inpainting, leveraging a foundation model, and ensuring temporal consistency and multi-view coherence.
Contribution
The paper presents a new 4D video creation method using a video inpainting foundation model, with a novel training strategy and inference modules for improved quality and editing capabilities.
Findings
Outperforms state-of-the-art in 4D video quality and versatility.
Enables prompt-based content editing with high consistency.
Effectively handles large camera motions during generation.
Abstract
We introduce Follow-Your-Creation, a novel 4D video creation framework capable of both generating and editing 4D content from a single monocular video input. By leveraging a powerful video inpainting foundation model as a generative prior, we reformulate 4D video creation as a video inpainting task, enabling the model to fill in missing content caused by camera trajectory changes or user edits. To facilitate this, we generate composite masked inpainting video data to effectively fine-tune the model for 4D video generation. Given an input video and its associated camera trajectory, we first perform depth-based point cloud rendering to obtain invisibility masks that indicate the regions that should be completed. Simultaneously, editing masks are introduced to specify user-defined modifications, and these are combined with the invisibility masks to create a composite masks dataset. During…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsGenerative Adversarial Networks and Image Synthesis · 3D Shape Modeling and Analysis · Face recognition and analysis
MethodsInpainting · Balanced Selection
