ModelScope Text-to-Video Technical Report

TL;DR

ModelScopeT2V is a novel text-to-video synthesis model that extends text-to-image techniques with spatio-temporal blocks, enabling consistent and smooth video generation adaptable to various datasets.

Contribution

It introduces a new text-to-video model with integrated spatio-temporal blocks and a large parameter count, improving over existing methods in quality and flexibility.

Findings

Outperforms state-of-the-art methods on key metrics

Supports variable frame numbers during training and inference

Demonstrates high-quality, smooth video synthesis

Abstract

This paper introduces ModelScopeT2V, a text-to-video synthesis model that evolves from a text-to-image synthesis model (i.e., Stable Diffusion). ModelScopeT2V incorporates spatio-temporal blocks to ensure consistent frame generation and smooth movement transitions. The model could adapt to varying frame numbers during training and inference, rendering it suitable for both image-text and video-text datasets. ModelScopeT2V brings together three components (i.e., VQGAN, a text encoder, and a denoising UNet), totally comprising 1.7 billion parameters, in which 0.5 billion parameters are dedicated to temporal capabilities. The model demonstrates superior performance over state-of-the-art methods across three evaluation metrics. The code and an online demo are available at \url{https://modelscope.cn/models/damo/text-to-video-synthesis/summary}.