Brain-Inspired Inference on Missing Video Sequence

TL;DR

This paper introduces a novel end-to-end neural architecture inspired by human inference to generate diverse plausible intermediate video sequences from two static frames, trained with adversarial unsupervised learning.

Contribution

The work presents a new model that learns to generate multiple plausible video sequences between two frames using latent variables and adversarial training.

Findings

Capable of generating diverse intermediate videos

Imitates human inference in video prediction

Effective on Moving MNIST and 2D Shapes datasets

Abstract

In this paper, we propose a novel end-to-end architecture that could generate a variety of plausible video sequences correlating two given discontinuous frames. Our work is inspired by the human ability of inference. Specifically, given two static images, human are capable of inferring what might happen in between as well as present diverse versions of their inference. We firstly train our model to learn the transformation to understand the movement trends within given frames. For the sake of imitating the inference of human, we introduce a latent variable sampled from Gaussian distribution. By means of integrating different latent variables with learned transformation features, the model could learn more various possible motion modes. Then applying these motion modes on the original frame, we could acquire various corresponding intermediate video sequence. Moreover, the framework is trained in adversarial fashion with unsupervised learning. Evaluating on the moving Mnist dataset and the 2D Shapes dataset, we show that our model is capable of imitating the human inference to some extent.