Stability-Informed Initialization of Neural Ordinary Differential Equations

TL;DR

This paper investigates how the stability regions of numerical integrators influence neural ODE training and introduces a stability-informed initialization method that improves performance across various benchmarks.

Contribution

It presents a novel initialization technique for neural ODEs based on stability analysis, enhancing training stability and prediction accuracy.

Findings

Stability regions of integrators affect neural ODE training.

The proposed initialization improves performance on benchmarks.

Stability-informed init reduces training instability.

Abstract

This paper addresses the training of Neural Ordinary Differential Equations (neural ODEs), and in particular explores the interplay between numerical integration techniques, stability regions, step size, and initialization techniques. It is shown how the choice of integration technique implicitly regularizes the learned model, and how the solver's corresponding stability region affects training and prediction performance. From this analysis, a stability-informed parameter initialization technique is introduced. The effectiveness of the initialization method is displayed across several learning benchmarks and industrial applications.