FlashVideo: A Framework for Swift Inference in Text-to-Video Generation

TL;DR

FlashVideo introduces a RetNet-based framework for text-to-video generation that significantly accelerates inference speed, making it practical for real-time applications by reducing complexity and improving efficiency.

Contribution

The paper presents the first adaptation of RetNet architecture for video generation, achieving faster inference with reduced complexity and a novel frame interpolation method.

Findings

Achieves 9.17x efficiency improvement over traditional autoregressive models

Reduces inference complexity from O(L^2) to O(L)

Inference speed comparable to BERT-based transformers

Abstract

In the evolving field of machine learning, video generation has witnessed significant advancements with autoregressive-based transformer models and diffusion models, known for synthesizing dynamic and realistic scenes. However, these models often face challenges with prolonged inference times, even for generating short video clips such as GIFs. This paper introduces FlashVideo, a novel framework tailored for swift Text-to-Video generation. FlashVideo represents the first successful adaptation of the RetNet architecture for video generation, bringing a unique approach to the field. Leveraging the RetNet-based architecture, FlashVideo reduces the time complexity of inference from $\mathcal{O}(L^2)$ to $\mathcal{O}(L)$ for a sequence of length $L$, significantly accelerating inference speed. Additionally, we adopt a redundant-free frame interpolation method, enhancing the efficiency of frame interpolation. Our comprehensive experiments demonstrate that FlashVideo achieves a $\times9.17$ efficiency improvement over a traditional autoregressive-based transformer model, and its inference speed is of the same order of magnitude as that of BERT-based transformer models.