UniPaint: Unified Space-time Video Inpainting via Mixture-of-Experts
Zhen Wan, Chenyang Qi, Zhiheng Liu, Tao Gui, Yue Ma

TL;DR
UniPaint introduces a unified framework for space-time video inpainting and interpolation, leveraging a Mixture of Experts and a novel masking strategy to enhance synthesis quality and versatility.
Contribution
It proposes a unified inpainting framework with a Mixture of Experts and a spatial-temporal masking strategy, enabling simultaneous inpainting and interpolation with mutual performance enhancement.
Findings
Achieves state-of-the-art results across multiple video inpainting and interpolation tasks.
Produces high-quality, aesthetically pleasing video results.
Demonstrates the effectiveness of the Mixture of Experts in handling diverse video synthesis tasks.
Abstract
In this paper, we present UniPaint, a unified generative space-time video inpainting framework that enables spatial-temporal inpainting and interpolation. Different from existing methods that treat video inpainting and video interpolation as two distinct tasks, we leverage a unified inpainting framework to tackle them and observe that these two tasks can mutually enhance synthesis performance. Specifically, we first introduce a plug-and-play space-time video inpainting adapter, which can be employed in various personalized models. The key insight is to propose a Mixture of Experts (MoE) attention to cover various tasks. Then, we design a spatial-temporal masking strategy during the training stage to mutually enhance each other and improve performance. UniPaint produces high-quality and aesthetically pleasing results, achieving the best quantitative results across various tasks and scale…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsGenerative Adversarial Networks and Image Synthesis · Advanced Image Processing Techniques · Image and Signal Denoising Methods
MethodsSoftmax · Attention Is All You Need · Inpainting
