COMUNI: Decomposing Common and Unique Video Signals for Diffusion-based Video Generation

TL;DR

COMUNI introduces a diffusion-based framework that decomposes common and unique video signals to improve the efficiency and quality of video generation, leveraging self-supervised learning and specialized diffusion streams.

Contribution

The paper proposes a novel method to decompose video signals into common and unique components, reducing computational complexity and enhancing generation quality in diffusion models.

Findings

Decomposition of video signals improves generation efficiency.

The proposed model outperforms existing methods in quality and speed.

Self-supervised training effectively learns signal decomposition.

Abstract

Since videos record objects moving coherently, adjacent video frames have commonness (similar object appearances) and uniqueness (slightly changed postures). To prevent redundant modeling of common video signals, we propose a novel diffusion-based framework, named COMUNI, which decomposes the COMmon and UNIque video signals to enable efficient video generation. Our approach separates the decomposition of video signals from the task of video generation, thus reducing the computation complexity of generative models. In particular, we introduce CU-VAE to decompose video signals and encode them into latent features. To train CU-VAE in a self-supervised manner, we employ a cascading merge module to reconstitute video signals and a time-agnostic video decoder to reconstruct video frames. Then we propose CU-LDM to model latent features for video generation, which adopts two specific diffusion streams to simultaneously model the common and unique latent features. We further utilize additional joint modules for cross modeling of the common and unique latent features, and a novel position embedding method to ensure the content consistency and motion coherence of generated videos. The position embedding method incorporates spatial and temporal absolute position information into the joint modules. Extensive experiments demonstrate the necessity of decomposing common and unique video signals for video generation and the effectiveness and efficiency of our proposed method.