Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model

TL;DR

This paper introduces Step-Video-T2V, a large-scale text-to-video model with advanced compression, bilingual encoding, and denoising techniques, achieving state-of-the-art quality on a new benchmark and discussing future challenges in video foundation modeling.

Contribution

The paper presents a novel 30B parameter text-to-video model with integrated compression, bilingual encoding, and a new evaluation benchmark, advancing the state-of-the-art in video generation.

Findings

Achieves state-of-the-art text-to-video quality on Step-Video-T2V-Eval benchmark.

Demonstrates effective video compression and high-quality reconstruction.

Identifies limitations and future directions for diffusion-based video models.

Abstract

We present Step-Video-T2V, a state-of-the-art text-to-video pre-trained model with 30B parameters and the ability to generate videos up to 204 frames in length. A deep compression Variational Autoencoder, Video-VAE, is designed for video generation tasks, achieving 16x16 spatial and 8x temporal compression ratios, while maintaining exceptional video reconstruction quality. User prompts are encoded using two bilingual text encoders to handle both English and Chinese. A DiT with 3D full attention is trained using Flow Matching and is employed to denoise input noise into latent frames. A video-based DPO approach, Video-DPO, is applied to reduce artifacts and improve the visual quality of the generated videos. We also detail our training strategies and share key observations and insights. Step-Video-T2V's performance is evaluated on a novel video generation benchmark, Step-Video-T2V-Eval, demonstrating its state-of-the-art text-to-video quality when compared with both open-source and commercial engines. Additionally, we discuss the limitations of current diffusion-based model paradigm and outline future directions for video foundation models. We make both Step-Video-T2V and Step-Video-T2V-Eval available at https://github.com/stepfun-ai/Step-Video-T2V. The online version can be accessed from https://yuewen.cn/videos as well. Our goal is to accelerate the innovation of video foundation models and empower video content creators.