Video prediction using score-based conditional density estimation

TL;DR

This paper introduces a score-based implicit density estimation framework for video prediction that effectively models uncertainty and handles occlusions, outperforming classical methods in selecting probable future frames.

Contribution

It presents a novel deep learning approach for conditional density estimation in video prediction, capable of managing high-dimensional data and ambiguous future states.

Findings

Handles occlusion boundaries better than classical methods

Automatically weights predictive evidence by reliability

Extracts adaptive representations for temporal prediction

Abstract

Temporal prediction is inherently uncertain, but representing the ambiguity in natural image sequences is a challenging high-dimensional probabilistic inference problem. For natural scenes, the curse of dimensionality renders explicit density estimation statistically and computationally intractable. Here, we describe an implicit regression-based framework for learning and sampling the conditional density of the next frame in a video given previous observed frames. We show that sequence-to-image deep networks trained on a simple resilience-to-noise objective function extract adaptive representations for temporal prediction. Synthetic experiments demonstrate that this score-based framework can handle occlusion boundaries: unlike classical methods that average over bifurcating temporal trajectories, it chooses among likely trajectories, selecting more probable options with higher frequency. Furthermore, analysis of networks trained on natural image sequences reveals that the representation automatically weights predictive evidence by its reliability, which is a hallmark of statistical inference