HARP: Autoregressive Latent Video Prediction with High-Fidelity Image Generator

TL;DR

HARP introduces a high-fidelity autoregressive latent video prediction model that leverages a VQ-GAN image generator and a causal transformer to produce high-resolution videos with improved quality and efficiency.

Contribution

The paper presents a scalable method combining a VQ-GAN generator with a causal transformer, achieving high-resolution video prediction with fewer parameters and minimal modifications to existing models.

Findings

Achieves competitive performance on standard benchmarks.

Produces high-resolution (256x256) videos.

Uses techniques like top-k sampling and data augmentation.

Abstract

Video prediction is an important yet challenging problem; burdened with the tasks of generating future frames and learning environment dynamics. Recently, autoregressive latent video models have proved to be a powerful video prediction tool, by separating the video prediction into two sub-problems: pre-training an image generator model, followed by learning an autoregressive prediction model in the latent space of the image generator. However, successfully generating high-fidelity and high-resolution videos has yet to be seen. In this work, we investigate how to train an autoregressive latent video prediction model capable of predicting high-fidelity future frames with minimal modification to existing models, and produce high-resolution (256x256) videos. Specifically, we scale up prior models by employing a high-fidelity image generator (VQ-GAN) with a causal transformer model, and introduce additional techniques of top-k sampling and data augmentation to further improve video prediction quality. Despite the simplicity, the proposed method achieves competitive performance to state-of-the-art approaches on standard video prediction benchmarks with fewer parameters, and enables high-resolution video prediction on complex and large-scale datasets. Videos are available at https://sites.google.com/view/harp-videos/home.