Image2Gif: Generating Continuous Realistic Animations with Warping NODEs

TL;DR

This paper introduces Warping Neural ODE, a novel framework for generating smooth, realistic animations from two frames by modeling continuous spatial transformations with differential equations, applicable in various training settings.

Contribution

It proposes a new continuous spatial transformation framework using Neural ODEs for realistic video frame interpolation between two images.

Findings

Achieves smooth, realistic animations with infinitesimal time steps.

Works effectively with GAN and L2 loss training methods.

Demonstrates applicability in generating animations from limited frames.

Abstract

Generating smooth animations from a limited number of sequential observations has a number of applications in vision. For example, it can be used to increase number of frames per second, or generating a new trajectory only based on first and last frames, e.g. a motion of face emotions. Despite the discrete observed data (frames), the problem of generating a new trajectory is a continues problem. In addition, to be perceptually realistic, the domain of an image should not alter drastically through the trajectory of changes. In this paper, we propose a new framework, Warping Neural ODE, for generating a smooth animation (video frame interpolation) in a continuous manner, given two ("farther apart") frames, denoting the start and the end of the animation. The key feature of our framework is utilizing the continuous spatial transformation of the image based on the vector field, derived from a system of differential equations. This allows us to achieve the smoothness and the realism of an animation with infinitely small time steps between the frames. We show the application of our work in generating an animation given two frames, in different training settings, including Generative Adversarial Network (GAN) and with $L_2$ loss.