Differentiable programming and its applications to dynamical systems

TL;DR

This paper introduces differentiable programming, combining neural networks and algorithms for modeling dynamical systems, highlighting its benefits, features like attention, and analyzing its advantages over traditional deep learning approaches.

Contribution

It presents the differentiable programming paradigm, discusses its features such as attention mechanisms, and reviews its applications and benefits in modeling dynamical systems.

Findings

Differentiable programming enhances modeling capabilities for dynamical systems.

Attention mechanisms improve focus and interpretability in models.

Differentiable models outperform traditional deep learning in certain dynamical tasks.

Abstract

Differentiable programming is the combination of classical neural networks modules with algorithmic ones in an end-to-end differentiable model. These new models, that use automatic differentiation to calculate gradients, have new learning capabilities (reasoning, attention and memory). In this tutorial, aimed at researchers in nonlinear systems with prior knowledge of deep learning, we present this new programming paradigm, describe some of its new features such as attention mechanisms, and highlight the benefits they bring. Then, we analyse the uses and limitations of traditional deep learning models in the modeling and prediction of dynamical systems. Here, a dynamical system is meant to be a set of state variables that evolve in time under general internal and external interactions. Finally, we review the advantages and applications of differentiable programming to dynamical systems.