FSVideo: Fast Speed Video Diffusion Model in a Highly-Compressed Latent Space

TL;DR

FSVideo introduces a fast transformer-based video diffusion framework utilizing a highly compressed latent space, a novel diffusion transformer architecture, and multi-resolution generation to achieve high-quality videos with significantly improved speed.

Contribution

The paper presents a new video autoencoder with a compressed latent space, a diffusion transformer with enhanced inter-layer communication, and a multi-resolution generation strategy, enabling faster video diffusion.

Findings

Achieves competitive video quality with much faster inference speed.

Uses a highly compressed latent space for efficient processing.

Demonstrates superior performance compared to existing open-source models.

Abstract

We introduce FSVideo, a fast speed transformer-based image-to-video (I2V) diffusion framework. We build our framework on the following key components: 1.) a new video autoencoder with highly-compressed latent space ($64\times64\times4$ spatial-temporal downsampling ratio), achieving competitive reconstruction quality; 2.) a diffusion transformer (DIT) architecture with a new layer memory design to enhance inter-layer information flow and context reuse within DIT, and 3.) a multi-resolution generation strategy via a few-step DIT upsampler to increase video fidelity. Our final model, which contains a 14B DIT base model and a 14B DIT upsampler, achieves competitive performance against other popular open-source models, while being an order of magnitude faster. We discuss our model design as well as training strategies in this report.