Dissecting Neural ODEs

TL;DR

This paper investigates Neural ODEs, aiming to understand their internal dynamics and how design choices influence their behavior, bridging deep learning and dynamical systems.

Contribution

It provides a detailed analysis of Neural ODEs' inner workings, enhancing interpretability and understanding of their dynamics.

Findings

Clarifies how design choices affect Neural ODE dynamics

Provides insights into the interpretability of continuous-depth models

Bridges the gap between deep learning and dynamical systems

Abstract

Continuous deep learning architectures have recently re-emerged as Neural Ordinary Differential Equations (Neural ODEs). This infinite-depth approach theoretically bridges the gap between deep learning and dynamical systems, offering a novel perspective. However, deciphering the inner working of these models is still an open challenge, as most applications apply them as generic black-box modules. In this work we "open the box", further developing the continuous-depth formulation with the aim of clarifying the influence of several design choices on the underlying dynamics.