Augmented Neural ODEs

TL;DR

Augmented Neural ODEs extend Neural ODEs by adding extra dimensions, enhancing expressiveness, stability, and generalization, while reducing computational costs, addressing key limitations of the original models.

Contribution

Introduction of Augmented Neural ODEs that improve upon Neural ODEs in expressiveness, stability, and efficiency, with theoretical and empirical validation.

Findings

Augmented Neural ODEs are more expressive than standard Neural ODEs.

They demonstrate better stability and generalization in experiments.

The models require lower computational costs.

Abstract

We show that Neural Ordinary Differential Equations (ODEs) learn representations that preserve the topology of the input space and prove that this implies the existence of functions Neural ODEs cannot represent. To address these limitations, we introduce Augmented Neural ODEs which, in addition to being more expressive models, are empirically more stable, generalize better and have a lower computational cost than Neural ODEs.