Neural Piecewise-Constant Delay Differential Equations

TL;DR

This paper introduces Neural Piecewise-Constant Delay Differential Equations (PCDDEs), a new continuous-depth neural network model that enhances modeling capabilities by incorporating multiple previous states, outperforming existing frameworks on various datasets.

Contribution

The paper proposes Neural PCDDEs, transforming delay into piecewise-constant delays, which improves modeling power without increasing network complexity.

Findings

Neural PCDDEs inherit universal approximation capabilities.

Neural PCDDEs outperform existing continuous-depth models on multiple datasets.

Effective in modeling delay population dynamics and real-world data.

Abstract

Continuous-depth neural networks, such as the Neural Ordinary Differential Equations (ODEs), have aroused a great deal of interest from the communities of machine learning and data science in recent years, which bridge the connection between deep neural networks and dynamical systems. In this article, we introduce a new sort of continuous-depth neural network, called the Neural Piecewise-Constant Delay Differential Equations (PCDDEs). Here, unlike the recently proposed framework of the Neural Delay Differential Equations (DDEs), we transform the single delay into the piecewise-constant delay(s). The Neural PCDDEs with such a transformation, on one hand, inherit the strength of universal approximating capability in Neural DDEs. On the other hand, the Neural PCDDEs, leveraging the contributions of the information from the multiple previous time steps, further promote the modeling capability without augmenting the network dimension. With such a promotion, we show that the Neural PCDDEs do outperform the several existing continuous-depth neural frameworks on the one-dimensional piecewise-constant delay population dynamics and real-world datasets, including MNIST, CIFAR10, and SVHN.