Towards End-to-End Generative Modeling of Long Videos with Memory-Efficient Bidirectional Transformers

TL;DR

This paper introduces Memory-efficient Bidirectional Transformer (MeBT), enabling efficient, parallel, end-to-end long video generation by capturing long-term dependencies with linear complexity, surpassing autoregressive models in quality and speed.

Contribution

The paper presents a novel bidirectional transformer architecture that achieves linear complexity for long video modeling, allowing fast, parallel decoding of entire videos from partial observations.

Findings

Significant improvement in video quality over autoregressive models

Faster inference times due to linear complexity

Effective modeling of long-term dependencies in videos

Abstract

Autoregressive transformers have shown remarkable success in video generation. However, the transformers are prohibited from directly learning the long-term dependency in videos due to the quadratic complexity of self-attention, and inherently suffering from slow inference time and error propagation due to the autoregressive process. In this paper, we propose Memory-efficient Bidirectional Transformer (MeBT) for end-to-end learning of long-term dependency in videos and fast inference. Based on recent advances in bidirectional transformers, our method learns to decode the entire spatio-temporal volume of a video in parallel from partially observed patches. The proposed transformer achieves a linear time complexity in both encoding and decoding, by projecting observable context tokens into a fixed number of latent tokens and conditioning them to decode the masked tokens through the cross-attention. Empowered by linear complexity and bidirectional modeling, our method demonstrates significant improvement over the autoregressive Transformers for generating moderately long videos in both quality and speed. Videos and code are available at https://sites.google.com/view/mebt-cvpr2023 .