Learning the Simplest Neural ODE

TL;DR

This paper investigates the training challenges of Neural ODEs using a simple linear model, introduces a stabilization method, and offers analytical insights to aid researchers in understanding and improving Neural ODE training.

Contribution

It presents a new stabilization technique for Neural ODEs and provides an analytical convergence analysis based on a simple linear model.

Findings

Training Neural ODEs is difficult due to inherent instability.

The proposed stabilization method improves training convergence.

Analytical insights help understand Neural ODE training dynamics.

Abstract

Since the advent of the ``Neural Ordinary Differential Equation (Neural ODE)'' paper, learning ODEs with deep learning has been applied to system identification, time-series forecasting, and related areas. Exploiting the diffeomorphic nature of ODE solution maps, neural ODEs has also enabled their use in generative modeling. Despite the rich potential to incorporate various kinds of physical information, training Neural ODEs remains challenging in practice. This study demonstrates, through the simplest one-dimensional linear model, why training Neural ODEs is difficult. We then propose a new stabilization method and provide an analytical convergence analysis. The insights and techniques presented here serve as a concise tutorial for researchers beginning work on Neural ODEs.