Deep Learning: Hydrodynamics, and Lie-Poisson Hamilton-Jacobi Theory

TL;DR

This paper presents a novel geometric framework for deep learning by modeling it as a hydrodynamics system with Lie-Poisson structure, enabling advanced analysis and structure-preserving numerical methods.

Contribution

It introduces a hydrodynamics and Lie-Poisson geometric formulation of deep learning, linking neural network training to quantum hydrodynamics and Schrödinger equations, and develops compatible numerical schemes.

Findings

Deep learning can be modeled as a hydrodynamics system with Lie-Poisson structure.

The training process relates to quantum hydrodynamics and nonlinear Schrödinger equations.

A structure-preserving numerical scheme is proposed for both deterministic and stochastic control.

Abstract

The interpretation of deep learning as a dynamical system has gained a considerable attention in recent years as it provides a promising framework. It allows for the use of existing ideas from established fields of mathematics for studying deep neural networks. In this article we present deep learning as an equivalent hydrodynamics formulation which in fact possess a Lie-Poisson structure and this further allows for using Poisson geometry to study deep learning. This is possible by considering the training of a deep neural network as a stochastic optimal control problem, which is solved using mean-field type control. The optimality conditions for the stochastic optimal control problem yield a system of partial differential equations, which we reduce to a system of equations of quantum hydrodynamics. We further take the hydrodynamics equivalence to show that, with conditions imposed on the network, that it is equivalent to the nonlinear Schr\"{o}dinger equation. As the such systems have a particular geometric structure, we also present a structure-preserving numerical scheme based on the Hamilton-Jacobi theory and its Lie-Poisson version known as Lie-Poisson Hamilton-Jacobi theory. The choice of this scheme was decided by the fact it can be applied to both deterministic and mean-field type control Hamiltonians. Not only this method eliminates the possibility of the emergence of fictitious solutions, it also eliminates the need for additional constraints when constructing high order methods.