xGen-VideoSyn-1: High-fidelity Text-to-Video Synthesis with Compressed Representations

TL;DR

xGen-VideoSyn-1 is a novel text-to-video synthesis model that uses compressed video representations and a divide-and-merge strategy to generate high-quality, long-duration videos efficiently from textual descriptions.

Contribution

The paper introduces VidVAE for video compression, a divide-and-merge approach for temporal consistency, and a Diffusion Transformer for robust video generation, advancing T2V capabilities.

Findings

Supports over 14-second 720p video generation

Achieves competitive performance against state-of-the-art models

Reduces computational costs with VidVAE and divide-and-merge strategy

Abstract

We present xGen-VideoSyn-1, a text-to-video (T2V) generation model capable of producing realistic scenes from textual descriptions. Building on recent advancements, such as OpenAI's Sora, we explore the latent diffusion model (LDM) architecture and introduce a video variational autoencoder (VidVAE). VidVAE compresses video data both spatially and temporally, significantly reducing the length of visual tokens and the computational demands associated with generating long-sequence videos. To further address the computational costs, we propose a divide-and-merge strategy that maintains temporal consistency across video segments. Our Diffusion Transformer (DiT) model incorporates spatial and temporal self-attention layers, enabling robust generalization across different timeframes and aspect ratios. We have devised a data processing pipeline from the very beginning and collected over 13M high-quality video-text pairs. The pipeline includes multiple steps such as clipping, text detection, motion estimation, aesthetics scoring, and dense captioning based on our in-house video-LLM model. Training the VidVAE and DiT models required approximately 40 and 642 H100 days, respectively. Our model supports over 14-second 720p video generation in an end-to-end way and demonstrates competitive performance against state-of-the-art T2V models.