Video Pixel Networks

TL;DR

The paper introduces Video Pixel Networks, a probabilistic model that captures the complex dependencies in video data, achieving state-of-the-art results on benchmarks and generating realistic videos with detailed motion.

Contribution

It presents a novel neural architecture that models the joint distribution of video pixels considering spatial, temporal, and color dependencies, advancing video generation techniques.

Findings

Achieves top performance on Moving MNIST benchmark.

Generates realistic videos with minor deviations from ground truth.

Generalizes well to new objects and actions in robotic pushing videos.

Abstract

We propose a probabilistic video model, the Video Pixel Network (VPN), that estimates the discrete joint distribution of the raw pixel values in a video. The model and the neural architecture reflect the time, space and color structure of video tensors and encode it as a four-dimensional dependency chain. The VPN approaches the best possible performance on the Moving MNIST benchmark, a leap over the previous state of the art, and the generated videos show only minor deviations from the ground truth. The VPN also produces detailed samples on the action-conditional Robotic Pushing benchmark and generalizes to the motion of novel objects.