DeNOTS: Stable Deep Neural ODEs for Time Series

TL;DR

DeNOTS introduces a novel method for stabilizing and deepening neural ODEs for time series, improving expressiveness and robustness without sacrificing stability, and demonstrates significant performance gains on multiple datasets.

Contribution

The paper proposes a new approach to increase neural ODE depth by scaling the integration horizon and stabilizing dynamics with Negative Feedback, ensuring stability and improved performance.

Findings

Outperforms existing models by up to 20% on four datasets

Provides theoretical bounds for Neural ODE risk using Gaussian process theory

Achieves a balance of expressiveness, stability, and robustness in continuous-time modeling

Abstract

Neural CDEs provide a natural way to process the temporal evolution of irregular time series. The number of function evaluations (NFE) is these systems' natural analog of depth (the number of layers in traditional neural networks). It is usually regulated via solver error tolerance: lower tolerance means higher numerical precision, requiring more integration steps. However, lowering tolerances does not adequately increase the models' expressiveness. We propose a simple yet effective alternative: scaling the integration time horizon to increase NFEs and "deepen`` the model. Increasing the integration interval causes uncontrollable growth in conventional vector fields, so we also propose a way to stabilize the dynamics via Negative Feedback (NF). It ensures provable stability without constraining flexibility. It also implies robustness: we provide theoretical bounds for Neural ODE risk using Gaussian process theory. Experiments on four open datasets demonstrate that our method, DeNOTS, outperforms existing approaches~ -- ~including recent Neural RDEs and state space models,~ -- ~achieving up to $20\%$ improvement in metrics. DeNOTS combines expressiveness, stability, and robustness, enabling reliable modelling in continuous-time domains.