InstructVid2Vid: Controllable Video Editing with Natural Language Instructions

TL;DR

InstructVid2Vid is a diffusion-based video editing method that allows natural language-guided modifications without fine-tuning, producing coherent and diverse videos efficiently.

Contribution

The paper introduces a novel end-to-end diffusion approach that enables controllable video editing guided by natural language instructions, using a new training dataset and coherence loss.

Findings

Produces high-quality, temporally coherent videos

Enables diverse edits like attribute, background, and style transfer

Eliminates need for per-example fine-tuning

Abstract

We introduce InstructVid2Vid, an end-to-end diffusion-based methodology for video editing guided by human language instructions. Our approach empowers video manipulation guided by natural language directives, eliminating the need for per-example fine-tuning or inversion. The proposed InstructVid2Vid model modifies a pretrained image generation model, Stable Diffusion, to generate a time-dependent sequence of video frames. By harnessing the collective intelligence of disparate models, we engineer a training dataset rich in video-instruction triplets, which is a more cost-efficient alternative to collecting data in real-world scenarios. To enhance the coherence between successive frames within the generated videos, we propose the Inter-Frames Consistency Loss and incorporate it during the training process. With multimodal classifier-free guidance during the inference stage, the generated videos is able to resonate with both the input video and the accompanying instructions. Experimental results demonstrate that InstructVid2Vid is capable of generating high-quality, temporally coherent videos and performing diverse edits, including attribute editing, background changes, and style transfer. These results underscore the versatility and effectiveness of our proposed method.