Symplectic Methods in Deep Learning

TL;DR

This paper introduces symplectic neural networks based on higher order explicit methods, combining theoretical stability guarantees with practical efficiency in modeling dynamical systems.

Contribution

It develops symplectic networks using higher order explicit methods that maintain non-vanishing gradients, enhancing stability and efficiency in learning dynamical systems.

Findings

Symplectic networks with higher order methods exhibit non-vanishing gradients.

The proposed architectures demonstrate improved efficiency on dynamical system tasks.

The approach combines theoretical guarantees with practical performance.

Abstract

Deep learning is widely used in tasks including image recognition and generation, in learning dynamical systems from data and many more. It is important to construct learning architectures with theoretical guarantees to permit safety in the applications. There has been considerable progress in this direction lately. In particular, symplectic networks were shown to have the non vanishing gradient property, essential for numerical stability. On the other hand, architectures based on higher order numerical methods were shown to be efficient in many tasks where the learned function has an underlying dynamical structure. In this work we construct symplectic networks based on higher order explicit methods with non vanishing gradient property and test their efficiency on various examples.