Simple Video Generation using Neural ODEs

TL;DR

This paper introduces a novel approach for video frame prediction using Neural ODEs to model continuous latent space dynamics, enabling extrapolation of future frames beyond training data.

Contribution

It proposes a new method that leverages Neural ODEs for continuous-time modeling in latent space for video generation, extending prior latent variable models.

Findings

Effective future frame prediction on Moving MNIST dataset.

Model demonstrates ability to extrapolate beyond trained time steps.

Promising results suggest potential for more advanced video generation tasks.

Abstract

Despite having been studied to a great extent, the task of conditional generation of sequences of frames, or videos, remains extremely challenging. It is a common belief that a key step towards solving this task resides in modelling accurately both spatial and temporal information in video signals. A promising direction to do so has been to learn latent variable models that predict the future in latent space and project back to pixels, as suggested in recent literature. Following this line of work and building on top of a family of models introduced in prior work, Neural ODE, we investigate an approach that models time-continuous dynamics over a continuous latent space with a differential equation with respect to time. The intuition behind this approach is that these trajectories in latent space could then be extrapolated to generate video frames beyond the time steps for which the model is trained. We show that our approach yields promising results in the task of future frame prediction on the Moving MNIST dataset with 1 and 2 digits.