Operator-learning-inspired Modeling of Neural Ordinary Differential Equations

TL;DR

This paper introduces a neural operator-based approach, called BFNO, for modeling the time-derivative in Neural ODEs, leading to improved performance across various tasks.

Contribution

It proposes a novel neural operator-inspired method for defining the time-derivative in NODEs, enhancing modeling capabilities over traditional neural network approaches.

Findings

BFNO significantly outperforms existing NODE methods in experiments.

The approach effectively models differential operators for downstream tasks.

Neural operators provide a promising direction for NODE modeling.

Abstract

Neural ordinary differential equations (NODEs), one of the most influential works of the differential equation-based deep learning, are to continuously generalize residual networks and opened a new field. They are currently utilized for various downstream tasks, e.g., image classification, time series classification, image generation, etc. Its key part is how to model the time-derivative of the hidden state, denoted dh(t)/dt. People have habitually used conventional neural network architectures, e.g., fully-connected layers followed by non-linear activations. In this paper, however, we present a neural operator-based method to define the time-derivative term. Neural operators were initially proposed to model the differential operator of partial differential equations (PDEs). Since the time-derivative of NODEs can be understood as a special type of the differential operator, our proposed method, called branched Fourier neural operator (BFNO), makes sense. In our experiments with general downstream tasks, our method significantly outperforms existing methods.