Predicting the Future with Transformational States

TL;DR

This paper introduces a neural network model that predicts future images by learning present states and their transformations, capturing realistic motion over multiple frames without adversarial training, and achieves competitive results on standard benchmarks.

Contribution

The paper presents a novel architecture combining latent state components with an RNN core for stable, multi-frame image prediction without adversarial training.

Findings

Generates stable, realistic motion sequences over multiple frames.

Achieves prediction accuracy comparable to state-of-the-art methods.

Operates effectively on benchmarks like Moving MNIST, KTH, and UCF101.

Abstract

An intelligent observer looks at the world and sees not only what is, but what is moving and what can be moved. In other words, the observer sees how the present state of the world can transform in the future. We propose a model that predicts future images by learning to represent the present state and its transformation given only a sequence of images. To do so, we introduce an architecture with a latent state composed of two components designed to capture (i) the present image state and (ii) the transformation between present and future states, respectively. We couple this latent state with a recurrent neural network (RNN) core that predicts future frames by transforming past states into future states by applying the accumulated state transformation with a learned operator. We describe how this model can be integrated into an encoder-decoder convolutional neural network (CNN) architecture that uses weighted residual connections to integrate representations of the past with representations of the future. Qualitatively, our approach generates image sequences that are stable and capture realistic motion over multiple predicted frames, without requiring adversarial training. Quantitatively, our method achieves prediction results comparable to state-of-the-art results on standard image prediction benchmarks (Moving MNIST, KTH, and UCF101).