On the Generalization and Approximation Capacities of Neural Controlled Differential Equations

TL;DR

This paper provides the first theoretical analysis of Neural Controlled Differential Equations (NCDEs), including generalization bounds and insights into how data irregularity impacts their performance, supported by experimental validation.

Contribution

It introduces the first theoretical framework for NCDEs, analyzing their generalization and approximation capacities in relation to data irregularity.

Findings

Generalization bound depends on data regularity

Continuity of CDE flow helps analyze bias

Classical neural network approximation results transfer to NCDEs

Abstract

Neural Controlled Differential Equations (NCDEs) are a state-of-the-art tool for supervised learning with irregularly sampled time series (Kidger, 2020). However, no theoretical analysis of their performance has been provided yet, and it remains unclear in particular how the irregularity of the time series affects their predictions. By merging the rich theory of controlled differential equations (CDE) and Lipschitz-based measures of the complexity of deep neural nets, we take a first step towards the theoretical understanding of NCDE. Our first result is a generalization bound for this class of predictors that depends on the regularity of the time series data. In a second time, we leverage the continuity of the flow of CDEs to provide a detailed analysis of both the sampling-induced bias and the approximation bias. Regarding this last result, we show how classical approximation results on neural nets may transfer to NCDEs. Our theoretical results are validated through a series of experiments.