CCEdit: Creative and Controllable Video Editing via Diffusion Models

TL;DR

CCEdit is a novel diffusion-based video editing framework that offers precise, creative, and controllable editing by separating structure and appearance controls, validated through extensive benchmarks and user studies.

Contribution

The paper introduces CCEdit, a new diffusion model-based framework with a trident network structure for versatile, controllable video editing, and a comprehensive benchmark dataset for evaluation.

Findings

CCEdit outperforms eight state-of-the-art methods in user studies.

The framework enables fine-grained control over structure and appearance.

The BalanceCC dataset facilitates thorough evaluation of video editing methods.

Abstract

In this paper, we present CCEdit, a versatile generative video editing framework based on diffusion models. Our approach employs a novel trident network structure that separates structure and appearance control, ensuring precise and creative editing capabilities. Utilizing the foundational ControlNet architecture, we maintain the structural integrity of the video during editing. The incorporation of an additional appearance branch enables users to exert fine-grained control over the edited key frame. These two side branches seamlessly integrate into the main branch, which is constructed upon existing text-to-image (T2I) generation models, through learnable temporal layers. The versatility of our framework is demonstrated through a diverse range of choices in both structure representations and personalized T2I models, as well as the option to provide the edited key frame. To facilitate comprehensive evaluation, we introduce the BalanceCC benchmark dataset, comprising 100 videos and 4 target prompts for each video. Our extensive user studies compare CCEdit with eight state-of-the-art video editing methods. The outcomes demonstrate CCEdit's substantial superiority over all other methods.