DIVE: Taming DINO for Subject-Driven Video Editing

TL;DR

DIVE leverages DINOv2 features to enable subject-driven, temporally consistent video editing guided by text prompts or reference images, advancing the quality and robustness of video editing techniques.

Contribution

The paper introduces DIVE, a novel framework that uses DINO features for improved subject-driven video editing with enhanced motion consistency and identity preservation.

Findings

Achieves high-quality, temporally consistent video edits.

Effectively registers target subjects using Low-Rank Adaptations.

Demonstrates robustness across diverse real-world videos.

Abstract

Building on the success of diffusion models in image generation and editing, video editing has recently gained substantial attention. However, maintaining temporal consistency and motion alignment still remains challenging. To address these issues, this paper proposes DINO-guided Video Editing (DIVE), a framework designed to facilitate subject-driven editing in source videos conditioned on either target text prompts or reference images with specific identities. The core of DIVE lies in leveraging the powerful semantic features extracted from a pretrained DINOv2 model as implicit correspondences to guide the editing process. Specifically, to ensure temporal motion consistency, DIVE employs DINO features to align with the motion trajectory of the source video. For precise subject editing, DIVE incorporates the DINO features of reference images into a pretrained text-to-image model to learn Low-Rank Adaptations (LoRAs), effectively registering the target subject's identity. Extensive experiments on diverse real-world videos demonstrate that our framework can achieve high-quality editing results with robust motion consistency, highlighting the potential of DINO to contribute to video editing. Project page: https://dino-video-editing.github.io