Learning to Take Directions One Step at a Time

TL;DR

This paper introduces a novel recurrent neural network architecture that generates animated video sequences from a single image guided by a sequence of motion strokes, enabling realistic and smooth animations.

Contribution

The paper presents a new recurrent architecture with explicit state separation and an autoencoding constraint for controllable video generation from a single image.

Findings

Capable of generating arbitrary-length animations from one image and motion sequence.

Uses autoencoding and GAN schemes for consistency and realism.

Effective on multiple datasets including MNIST and Human3.6M.

Abstract

We present a method to generate a video sequence given a single image. Because items in an image can be animated in arbitrarily many different ways, we introduce as control signal a sequence of motion strokes. Such control signal can be automatically transferred from other videos, e.g., via bounding box tracking. Each motion stroke provides the direction to the moving object in the input image and we aim to train a network to generate an animation following a sequence of such directions. To address this task we design a novel recurrent architecture, which can be trained easily and effectively thanks to an explicit separation of past, future and current states. As we demonstrate in the experiments, our proposed architecture is capable of generating an arbitrary number of frames from a single image and a sequence of motion strokes. Key components of our architecture are an autoencoding constraint to ensure consistency with the past and a generative adversarial scheme to ensure that images look realistic and are temporally smooth. We demonstrate the effectiveness of our approach on the MNIST, KTH, Human3.6M, Push and Weizmann datasets.