UniPaint: Unified Space-time Video Inpainting via Mixture-of-Experts

TL;DR

UniPaint introduces a unified framework for space-time video inpainting and interpolation, leveraging a Mixture of Experts and a novel masking strategy to enhance synthesis quality and versatility.

Contribution

It proposes a unified inpainting framework with a Mixture of Experts and a spatial-temporal masking strategy, enabling simultaneous inpainting and interpolation with mutual performance enhancement.

Findings

Achieves state-of-the-art results across multiple video inpainting and interpolation tasks.

Produces high-quality, aesthetically pleasing video results.

Demonstrates the effectiveness of the Mixture of Experts in handling diverse video synthesis tasks.

Abstract

In this paper, we present UniPaint, a unified generative space-time video inpainting framework that enables spatial-temporal inpainting and interpolation. Different from existing methods that treat video inpainting and video interpolation as two distinct tasks, we leverage a unified inpainting framework to tackle them and observe that these two tasks can mutually enhance synthesis performance. Specifically, we first introduce a plug-and-play space-time video inpainting adapter, which can be employed in various personalized models. The key insight is to propose a Mixture of Experts (MoE) attention to cover various tasks. Then, we design a spatial-temporal masking strategy during the training stage to mutually enhance each other and improve performance. UniPaint produces high-quality and aesthetically pleasing results, achieving the best quantitative results across various tasks and scale setups. The code and checkpoints are available at $\href{https://github.com/mmmmm-w/UniPaint}{this \ repository}$.