Neural Differential Recurrent Neural Network with Adaptive Time Steps

TL;DR

This paper introduces RNN-ODE-Adap, an adaptive time step neural ODE model for non-stationary, spike-like time series, improving prediction accuracy and efficiency over existing methods.

Contribution

The paper presents a novel RNN-ODE model with adaptive time steps, providing theoretical guarantees and demonstrating superior performance on various datasets.

Findings

Higher prediction accuracy on simulated and real data

Reduced computational cost due to adaptive stepping

Provable consistency for Hawkes-type processes

Abstract

The neural Ordinary Differential Equation (ODE) model has shown success in learning complex continuous-time processes from observations on discrete time stamps. In this work, we consider the modeling and forecasting of time series data that are non-stationary and may have sharp changes like spikes. We propose an RNN-based model, called RNN-ODE-Adap, that uses a neural ODE to represent the time development of the hidden states, and we adaptively select time steps based on the steepness of changes of the data over time so as to train the model more efficiently for the "spike-like" time series. Theoretically, RNN-ODE-Adap yields provably a consistent estimation of the intensity function for the Hawkes-type time series data. We also provide an approximation analysis of the RNN-ODE model showing the benefit of adaptive steps. The proposed model is demonstrated to achieve higher prediction accuracy with reduced computational cost on simulated dynamic system data and point process data and on a real electrocardiography dataset.