Enhancing Video Inpainting with Aligned Frame Interval Guidance

TL;DR

VidPivot introduces a novel video inpainting framework that utilizes aligned frame interval guidance and a content propagation module to improve spatiotemporal coherence and reduce content degradation.

Contribution

The paper proposes VidPivot, which decouples video inpainting into image inpainting and motion propagation, using frame interval priors and a dedicated context controller for enhanced coherence.

Findings

Achieves competitive performance on multiple benchmarks.

Effectively reduces content degradation within video chunks.

Generalizes well across various video inpainting scenarios.

Abstract

Recent image-to-video (I2V) based video inpainting methods have made significant strides by leveraging single-image priors and modeling temporal consistency across masked frames. Nevertheless, these methods suffer from severe content degradation within video chunks. Furthermore, the absence of a robust frame alignment scheme compromises intra-chunk and inter-chunk spatiotemporal stability, resulting in insufficient control over the entire video. To address these limitations, we propose VidPivot, a novel framework that decouples video inpainting into two sub-tasks: multi-frame consistent image inpainting and masked area motion propagation. Our approach introduces frame interval priors as spatiotemporal cues to guide the inpainting process. To enhance cross-frame coherence, we design a FrameProp Module that implements a frame content propagation strategy, diffusing reference frame content into subsequent frames via a splicing mechanism. Additionally, a dedicated context controller encodes these coherent frame priors into the I2V generative backbone, effectively serving as soft constrain to suppress content distortion during generation. Extensive evaluations demonstrate that VidPivot achieves competitive performance across diverse benchmarks and generalizes well to different video inpainting scenarios.