Kiwi-Edit: Versatile Video Editing via Instruction and Reference Guidance

TL;DR

Kiwi-Edit introduces a scalable data generation pipeline and a unified architecture for instruction and reference-guided video editing, achieving state-of-the-art results in controllable editing tasks.

Contribution

The paper presents a novel data synthesis pipeline, a large-scale dataset, and a unified editing model that significantly improve instruction and reference-guided video editing.

Findings

Achieved state-of-the-art performance on instruction-reference-following tasks.

Developed a scalable pipeline to generate high-quality training data.

Proposed a unified architecture that enhances reference fidelity and instruction following.

Abstract

Instruction-based video editing has witnessed rapid progress, yet current methods often struggle with precise visual control, as natural language is inherently limited in describing complex visual nuances. Although reference-guided editing offers a robust solution, its potential is currently bottlenecked by the scarcity of high-quality paired training data. To bridge this gap, we introduce a scalable data generation pipeline that transforms existing video editing pairs into high-fidelity training quadruplets, leveraging image generative models to create synthesized reference scaffolds. Using this pipeline, we construct RefVIE, a large-scale dataset tailored for instruction-reference-following tasks, and establish RefVIE-Bench for comprehensive evaluation. Furthermore, we propose a unified editing architecture, Kiwi-Edit, that synergizes learnable queries and latent visual features for reference semantic guidance. Our model achieves significant gains in instruction following and reference fidelity via a progressive multi-stage training curriculum. Extensive experiments demonstrate that our data and architecture establish a new state-of-the-art in controllable video editing. All datasets, models, and code is released at https://github.com/showlab/Kiwi-Edit.