Object-Centric Diffusion for Efficient Video Editing

TL;DR

This paper introduces Object-Centric Diffusion, a method that significantly speeds up video editing by focusing computational resources on important regions, reducing latency up to 10 times while maintaining quality.

Contribution

It proposes two novel techniques, Object-Centric Sampling and Token Merging, that improve efficiency and artifact correction in diffusion-based video editing without retraining models.

Findings

Achieves up to 10x latency reduction with comparable quality.

Effectively reduces memory and computational costs.

Applicable to existing models without retraining.

Abstract

Diffusion-based video editing have reached impressive quality and can transform either the global style, local structure, and attributes of given video inputs, following textual edit prompts. However, such solutions typically incur heavy memory and computational costs to generate temporally-coherent frames, either in the form of diffusion inversion and/or cross-frame attention. In this paper, we conduct an analysis of such inefficiencies, and suggest simple yet effective modifications that allow significant speed-ups whilst maintaining quality. Moreover, we introduce Object-Centric Diffusion, to fix generation artifacts and further reduce latency by allocating more computations towards foreground edited regions, arguably more important for perceptual quality. We achieve this by two novel proposals: i) Object-Centric Sampling, decoupling the diffusion steps spent on salient or background regions and spending most on the former, and ii) Object-Centric Token Merging, which reduces cost of cross-frame attention by fusing redundant tokens in unimportant background regions. Both techniques are readily applicable to a given video editing model without retraining, and can drastically reduce its memory and computational cost. We evaluate our proposals on inversion-based and control-signal-based editing pipelines, and show a latency reduction up to 10x for a comparable synthesis quality. Project page: qualcomm-ai-research.github.io/object-centric-diffusion.