Controllable Video Generation with Provable Disentanglement

TL;DR

This paper introduces CoVoGAN, a novel framework for controllable video generation that disentangles static and dynamic concepts, providing precise control and theoretical guarantees of identifiability, validated through extensive experiments.

Contribution

We propose a theoretically grounded method for disentangling video concepts, enabling independent control over static and dynamic features in generated videos.

Findings

Improved control precision in video generation.

Theoretical proof of latent variable identifiability.

Enhanced generation quality demonstrated on benchmarks.

Abstract

Controllable video generation remains a significant challenge, despite recent advances in generating high-quality and consistent videos. Most existing methods for controlling video generation treat the video as a whole, neglecting intricate fine-grained spatiotemporal relationships, which limits both control precision and efficiency. In this paper, we propose Controllable Video Generative Adversarial Networks (CoVoGAN) to disentangle the video concepts, thus facilitating efficient and independent control over individual concepts. Specifically, following the minimal change principle, we first disentangle static and dynamic latent variables. We then leverage the sufficient change property to achieve component-wise identifiability of dynamic latent variables, enabling disentangled control of video generation. To establish the theoretical foundation, we provide a rigorous analysis demonstrating the identifiability of our approach. Building on these theoretical insights, we design a Temporal Transition Module to disentangle latent dynamics. To enforce the minimal change principle and sufficient change property, we minimize the dimensionality of latent dynamic variables and impose temporal conditional independence. To validate our approach, we integrate this module as a plug-in for GANs. Extensive qualitative and quantitative experiments on various video generation benchmarks demonstrate that our method significantly improves generation quality and controllability across diverse real-world scenarios.