VIRES: Video Instance Repainting via Sketch and Text Guided Generation

TL;DR

VIRES is a novel video instance repainting method that uses sketch and text guidance to achieve high-quality, temporally consistent video editing, outperforming existing approaches.

Contribution

Introduces VIRES, a new framework leveraging generative priors, a Sequential ControlNet, and a sketch-aware encoder for improved video instance editing.

Findings

Outperforms state-of-the-art in visual quality and temporal consistency

Achieves better alignment with sketch and text conditions

Demonstrates effectiveness on the VireSet dataset

Abstract

We introduce VIRES, a video instance repainting method with sketch and text guidance, enabling video instance repainting, replacement, generation, and removal. Existing approaches struggle with temporal consistency and accurate alignment with the provided sketch sequence. VIRES leverages the generative priors of text-to-video models to maintain temporal consistency and produce visually pleasing results. We propose the Sequential ControlNet with the standardized self-scaling, which effectively extracts structure layouts and adaptively captures high-contrast sketch details. We further augment the diffusion transformer backbone with the sketch attention to interpret and inject fine-grained sketch semantics. A sketch-aware encoder ensures that repainted results are aligned with the provided sketch sequence. Additionally, we contribute the VireSet, a dataset with detailed annotations tailored for training and evaluating video instance editing methods. Experimental results demonstrate the effectiveness of VIRES, which outperforms state-of-the-art methods in visual quality, temporal consistency, condition alignment, and human ratings. Project page: https://hjzheng.net/projects/VIRES/