REDUCIO! Generating 1K Video within 16 Seconds using Extremely Compressed Motion Latents

TL;DR

This paper introduces Reducio, a highly compressed video encoding method that drastically reduces latents, enabling fast high-resolution video generation with minimal computational resources.

Contribution

We propose Reducio, an extremely compressed latent space for videos, enabling efficient high-resolution video generation with minimal loss of quality.

Findings

64x reduction in latents compared to standard VAE

Generates 16-frame 1024x1024 videos in 15.5 seconds

Reduces training GPU hours significantly

Abstract

Commercial video generation models have exhibited realistic, high-fidelity results but are still restricted to limited access. One crucial obstacle for large-scale applications is the expensive training and inference cost. In this paper, we argue that videos contain significantly more redundant information than images, allowing them to be encoded with very few motion latents. Towards this goal, we design an image-conditioned VAE that projects videos into extremely compressed latent space and decode them based on content images. This magic Reducio charm enables 64x reduction of latents compared to a common 2D VAE, without sacrificing the quality. Building upon Reducio-VAE, we can train diffusion models for high-resolution video generation efficiently. Specifically, we adopt a two-stage generation paradigm, first generating a condition image via text-to-image generation, followed by text-image-to-video generation with the proposed Reducio-DiT. Extensive experiments show that our model achieves strong performance in evaluation. More importantly, our method significantly boosts the training and inference efficiency of video LDMs. Reducio-DiT is trained in just 3.2K A100 GPU hours in total and can generate a 16-frame 1024$\times$1024 video clip within 15.5 seconds on a single A100 GPU. Code released at https://github.com/microsoft/Reducio-VAE .