Predicting Video with VQVAE

TL;DR

This paper introduces a novel video prediction method using VQ-VAE to compress videos into discrete latent variables, enabling high-resolution prediction on large-scale, diverse datasets like Kinetics-600.

Contribution

The paper presents a new approach combining VQ-VAE with autoregressive models for scalable, high-resolution video prediction on unconstrained datasets.

Findings

Achieved high-resolution (256x256) video prediction on large-scale datasets.

Outperformed previous methods in diversity and resolution.

Validated results with crowdsourced human evaluation.

Abstract

In recent years, the task of video prediction-forecasting future video given past video frames-has attracted attention in the research community. In this paper we propose a novel approach to this problem with Vector Quantized Variational AutoEncoders (VQ-VAE). With VQ-VAE we compress high-resolution videos into a hierarchical set of multi-scale discrete latent variables. Compared to pixels, this compressed latent space has dramatically reduced dimensionality, allowing us to apply scalable autoregressive generative models to predict video. In contrast to previous work that has largely emphasized highly constrained datasets, we focus on very diverse, large-scale datasets such as Kinetics-600. We predict video at a higher resolution on unconstrained videos, 256x256, than any other previous method to our knowledge. We further validate our approach against prior work via a crowdsourced human evaluation.