A unified framework for Hamiltonian deep neural networks

TL;DR

This paper introduces a Hamiltonian-based framework for deep neural networks that enhances stability and expressiveness, leading to improved training and performance on benchmark classification tasks like MNIST.

Contribution

It proposes a novel Hamiltonian deep neural network framework that ensures marginal stability and derives new architectures with better training dynamics.

Findings

Hamiltonian DNNs exhibit marginal stability, reducing gradient vanishing/explosion.

The framework encompasses existing models and enables new, more expressive architectures.

Experimental results show improved performance on MNIST classification.

Abstract

Training deep neural networks (DNNs) can be difficult due to the occurrence of vanishing/exploding gradients during weight optimization. To avoid this problem, we propose a class of DNNs stemming from the time discretization of Hamiltonian systems. The time-invariant version of the corresponding Hamiltonian models enjoys marginal stability, a property that, as shown in previous works and for specific DNNs architectures, can mitigate convergence to zero or divergence of gradients. In the present paper, we formally study this feature by deriving and analysing the backward gradient dynamics in continuous time. The proposed Hamiltonian framework, besides encompassing existing networks inspired by marginally stable ODEs, allows one to derive new and more expressive architectures. The good performance of the novel DNNs is demonstrated on benchmark classification problems, including digit recognition using the MNIST dataset.