A Hierarchical Variational Neural Uncertainty Model for Stochastic Video Prediction

TL;DR

This paper introduces a hierarchical variational neural model that quantifies predictive uncertainty to improve stochastic video prediction, especially with limited training data, leading to better quality and diversity in generated videos.

Contribution

It proposes a novel Neural Uncertainty Quantifier (NUQ) within a hierarchical Bayesian framework to enhance training and prediction in stochastic video modeling.

Findings

Outperforms state-of-the-art models on benchmark datasets.

Improves training effectiveness with small datasets.

Generates higher quality and more diverse videos.

Abstract

Predicting the future frames of a video is a challenging task, in part due to the underlying stochastic real-world phenomena. Prior approaches to solve this task typically estimate a latent prior characterizing this stochasticity, however do not account for the predictive uncertainty of the (deep learning) model. Such approaches often derive the training signal from the mean-squared error (MSE) between the generated frame and the ground truth, which can lead to sub-optimal training, especially when the predictive uncertainty is high. Towards this end, we introduce Neural Uncertainty Quantifier (NUQ) - a stochastic quantification of the model's predictive uncertainty, and use it to weigh the MSE loss. We propose a hierarchical, variational framework to derive NUQ in a principled manner using a deep, Bayesian graphical model. Our experiments on four benchmark stochastic video prediction datasets show that our proposed framework trains more effectively compared to the state-of-the-art models (especially when the training sets are small), while demonstrating better video generation quality and diversity against several evaluation metrics.