MagicVideo: Efficient Video Generation With Latent Diffusion Models

TL;DR

MagicVideo introduces an efficient text-to-video generation framework using latent diffusion models, achieving high-quality video synthesis with significantly reduced computational costs and novel design adaptations for video data.

Contribution

It proposes a novel latent diffusion-based approach with a 3D U-Net design and new modules for efficient, high-quality video generation from text descriptions.

Findings

Generates smooth, high-quality videos aligned with text prompts

Achieves 64x reduction in FLOPs compared to existing video diffusion models

Produces videos at 256x256 resolution on a single GPU

Abstract

We present an efficient text-to-video generation framework based on latent diffusion models, termed MagicVideo. MagicVideo can generate smooth video clips that are concordant with the given text descriptions. Due to a novel and efficient 3D U-Net design and modeling video distributions in a low-dimensional space, MagicVideo can synthesize video clips with 256x256 spatial resolution on a single GPU card, which takes around 64x fewer computations than the Video Diffusion Models (VDM) in terms of FLOPs. In specific, unlike existing works that directly train video models in the RGB space, we use a pre-trained VAE to map video clips into a low-dimensional latent space and learn the distribution of videos' latent codes via a diffusion model. Besides, we introduce two new designs to adapt the U-Net denoiser trained on image tasks to video data: a frame-wise lightweight adaptor for the image-to-video distribution adjustment and a directed temporal attention module to capture temporal dependencies across frames. Thus, we can exploit the informative weights of convolution operators from a text-to-image model for accelerating video training. To ameliorate the pixel dithering in the generated videos, we also propose a novel VideoVAE auto-encoder for better RGB reconstruction. We conduct extensive experiments and demonstrate that MagicVideo can generate high-quality video clips with either realistic or imaginary content. Refer to \url{https://magicvideo.github.io/#} for more examples.