Learning Dynamic Generator Model by Alternating Back-Propagation Through Time

TL;DR

This paper introduces a dynamic generator model for spatial-temporal data like videos, using an auto-regressive latent state sequence and an alternating back-propagation algorithm for training.

Contribution

It proposes a novel dynamic generator model with a non-linear auto-regressive latent process and a new training method, alternating back-propagation through time.

Findings

Successfully models dynamic textures and action sequences

Demonstrates realistic generation of video data

Provides an effective training algorithm for complex temporal models

Abstract

This paper studies the dynamic generator model for spatial-temporal processes such as dynamic textures and action sequences in video data. In this model, each time frame of the video sequence is generated by a generator model, which is a non-linear transformation of a latent state vector, where the non-linear transformation is parametrized by a top-down neural network. The sequence of latent state vectors follows a non-linear auto-regressive model, where the state vector of the next frame is a non-linear transformation of the state vector of the current frame as well as an independent noise vector that provides randomness in the transition. The non-linear transformation of this transition model can be parametrized by a feedforward neural network. We show that this model can be learned by an alternating back-propagation through time algorithm that iteratively samples the noise vectors and updates the parameters in the transition model and the generator model. We show that our training method can learn realistic models for dynamic textures and action patterns.