Efficient Neural Controlled Differential Equations via Attentive Kernel Smoothing

TL;DR

This paper introduces a novel kernel and Gaussian Process smoothing approach for Neural CDEs, improving efficiency and accuracy by controlling trajectory regularity and employing attention-based multi-view reconstruction.

Contribution

It proposes a new path construction method for Neural CDEs using kernel and GP smoothing, along with an attention-based multi-view approach for detail recovery, enhancing efficiency and expressiveness.

Findings

Achieves state-of-the-art accuracy on sequence modeling tasks.

Reduces Number of Function Evaluations (NFE) and inference time.

Outperforms spline-based methods in efficiency and accuracy.

Abstract

Neural Controlled Differential Equations (Neural CDEs) provide a powerful continuous-time framework for sequence modeling, yet the roughness of the driving control path often restricts their efficiency. Standard splines introduce high-frequency variations that force adaptive solvers to take excessively small steps, driving up the Number of Function Evaluations (NFE). We propose a novel approach to Neural CDE path construction that replaces exact interpolation with Kernel and Gaussian Process (GP) smoothing, enabling explicit control over trajectory regularity. To recover details lost during smoothing, we propose an attention-based Multi-View CDE (MV-CDE) and its convolutional extension (MVC-CDE), which employ learnable queries to inform path reconstruction. This framework allows the model to distribute representational capacity across multiple trajectories, each capturing distinct temporal patterns. Empirical results demonstrate that our method, MVC-CDE with GP, achieves state-of-the-art accuracy while significantly reducing NFEs and total inference time compared to spline-based baselines.