From Here to There: Video Inbetweening Using Direct 3D Convolutions

TL;DR

This paper introduces a fully convolutional model for generating diverse in-between video sequences from start and end frames, using 3D convolutions and adversarial training for improved realism and diversity.

Contribution

It proposes a novel convolutional approach with a stochastic fusion mechanism for inbetweening, avoiding recurrent networks and enabling end-to-end training.

Findings

Generates meaningful and diverse video sequences

Outperforms existing methods on benchmark datasets

Achieves realistic in-between frames through adversarial training

Abstract

We consider the problem of generating plausible and diverse video sequences, when we are only given a start and an end frame. This task is also known as inbetweening, and it belongs to the broader area of stochastic video generation, which is generally approached by means of recurrent neural networks (RNN). In this paper, we propose instead a fully convolutional model to generate video sequences directly in the pixel domain. We first obtain a latent video representation using a stochastic fusion mechanism that learns how to incorporate information from the start and end frames. Our model learns to produce such latent representation by progressively increasing the temporal resolution, and then decode in the spatiotemporal domain using 3D convolutions. The model is trained end-to-end by minimizing an adversarial loss. Experiments on several widely-used benchmark datasets show that it is able to generate meaningful and diverse in-between video sequences, according to both quantitative and qualitative evaluations.