Differentiable Multiple Shooting Layers

TL;DR

This paper introduces Differentiable Multiple Shooting Layers (MSLs), a new class of implicit neural models that improve efficiency and parallelization in solving differential equations, with applications in control, sequence generation, and time series classification.

Contribution

The paper presents the algorithmic framework for MSLs, analyzing their theoretical and computational advantages over Neural ODEs, and demonstrates their effectiveness in various complex tasks.

Findings

MSLs reduce the number of function evaluations (NFEs).

MSLs achieve faster wall-clock inference times.

MSLs perform well in long horizon control and time series classification.

Abstract

We detail a novel class of implicit neural models. Leveraging time-parallel methods for differential equations, Multiple Shooting Layers (MSLs) seek solutions of initial value problems via parallelizable root-finding algorithms. MSLs broadly serve as drop-in replacements for neural ordinary differential equations (Neural ODEs) with improved efficiency in number of function evaluations (NFEs) and wall-clock inference time. We develop the algorithmic framework of MSLs, analyzing the different choices of solution methods from a theoretical and computational perspective. MSLs are showcased in long horizon optimal control of ODEs and PDEs and as latent models for sequence generation. Finally, we investigate the speedups obtained through application of MSL inference in neural controlled differential equations (Neural CDEs) for time series classification of medical data.