Vid-ODE: Continuous-Time Video Generation with Neural Ordinary Differential Equation

TL;DR

Vid-ODE introduces a novel continuous-time video generation framework using neural ODEs, enabling flexible frame rate handling and high-quality frame synthesis for real-world videos.

Contribution

It is the first to successfully apply neural ODEs to continuous-time video generation, improving flexibility and quality over fixed-frame-rate models.

Findings

Outperforms state-of-the-art methods in various settings

Handles both interpolation and extrapolation tasks

Maintains frame sharpness with pixel-level techniques

Abstract

Video generation models often operate under the assumption of fixed frame rates, which leads to suboptimal performance when it comes to handling flexible frame rates (e.g., increasing the frame rate of the more dynamic portion of the video as well as handling missing video frames). To resolve the restricted nature of existing video generation models' ability to handle arbitrary timesteps, we propose continuous-time video generation by combining neural ODE (Vid-ODE) with pixel-level video processing techniques. Using ODE-ConvGRU as an encoder, a convolutional version of the recently proposed neural ODE, which enables us to learn continuous-time dynamics, Vid-ODE can learn the spatio-temporal dynamics of input videos of flexible frame rates. The decoder integrates the learned dynamics function to synthesize video frames at any given timesteps, where the pixel-level composition technique is used to maintain the sharpness of individual frames. With extensive experiments on four real-world video datasets, we verify that the proposed Vid-ODE outperforms state-of-the-art approaches under various video generation settings, both within the trained time range (interpolation) and beyond the range (extrapolation). To the best of our knowledge, Vid-ODE is the first work successfully performing continuous-time video generation using real-world videos.