Multisymplectic Formulation of Deep Learning Using Mean--Field Type Control and Nonlinear Stability of Training Algorithm

TL;DR

This paper introduces a multisymplectic geometric framework for deep learning by modeling training as a hydrodynamics system, enabling stability analysis and numerical solutions based on mean-field control and geometric structures.

Contribution

It formulates deep neural network training as a multisymplectic hydrodynamics system using mean-field control, providing new insights into stability and geometric numerical schemes.

Findings

Numerical scheme preserves geometric structure of the hydrodynamics system.

Stability conditions guide the selection of network architecture.

Framework links deep learning with multisymplectic geometry and fluid dynamics.

Abstract

As it stands, a robust mathematical framework to analyse and study various topics in deep learning is yet to come to the fore. Nonetheless, viewing deep learning as a dynamical system allows the use of established theories to investigate the behaviour of deep neural networks. In order to study the stability of the training process, in this article, we formulate training of deep neural networks as a hydrodynamics system, which has a multisymplectic structure. For that, the deep neural network is modelled using a stochastic differential equation and, thereby, mean-field type control is used to train it. The necessary conditions for optimality of the mean--field type control reduce to a system of Euler-Poincare equations, which has the a similar geometric structure to that of compressible fluids. The mean-field type control is solved numerically using a multisymplectic numerical scheme that takes advantage of the underlying geometry. Moreover, the numerical scheme, yields an approximated solution which is also an exact solution of a hydrodynamics system with a multisymplectic structure and it can be analysed using backward error analysis. Further, nonlinear stability yields the condition for selecting the number of hidden layers and the number of nodes per layer, that makes the training stable while approximating the solution of a residual neural network with a number of hidden layers approaching infinity.