InsEdit: Towards Instruction-based Visual Editing via Data-Efficient Video Diffusion Models Adaptation

TL;DR

InsEdit is an instruction-based video editing model that achieves state-of-the-art results with minimal data by leveraging a novel architecture and training pipeline, also supporting image editing.

Contribution

It introduces InsEdit, a data-efficient video editing model built on HunyuanVideo-1.5, utilizing Mutual Context Attention for flexible editing from limited data.

Findings

Achieves state-of-the-art results with only ~100K video editing data.

Supports image editing without additional training.

Uses Mutual Context Attention to enable edits starting mid-clip.

Abstract

Instruction-based video editing is a natural way to control video content with text, but adapting a video generation model into an editor usually appears data-hungry. At the same time, high-quality video editing data remains scarce. In this paper, we show that a video generation backbone can become a strong video editor without large scale video editing data. We present InsEdit, an instruction-based editing model built on HunyuanVideo-1.5. InsEdit combines a visual editing architecture with a video data pipeline based on Mutual Context Attention (MCA), which creates aligned video pairs where edits can begin in the middle of a clip rather than only from the first frame. With only O(100)K video editing data, InsEdit achieves state-of-the-art results among open-source methods on our video instruction editing benchmarks. In addition, because our training recipe also includes image editing data, the final model supports image editing without any modification.