Constructing Gradient Controllable Recurrent Neural Networks Using Hamiltonian Dynamics

TL;DR

This paper introduces Hamiltonian RNNs, a novel architecture inspired by Hamiltonian dynamics, which effectively controls gradients over long sequences, outperforming existing RNNs without extensive hyperparameter tuning.

Contribution

It proposes a Hamiltonian-based RNN architecture that inherits long-term stability properties from dynamical systems, enabling better gradient control in sequential learning.

Findings

Hamiltonian RNNs control gradients over long sequences.

They outperform state-of-the-art RNNs in simulations.

Hyperparameter heuristic simplifies model tuning.

Abstract

Recurrent neural networks (RNNs) have gained a great deal of attention in solving sequential learning problems. The learning of long-term dependencies, however, remains challenging due to the problem of a vanishing or exploding hidden states gradient. By exploring further the recently established connections between RNNs and dynamical systems we propose a novel RNN architecture, which we call a Hamiltonian recurrent neural network (Hamiltonian RNN), based on a symplectic discretization of an appropriately chosen Hamiltonian system. The key benefit of this approach is that the corresponding RNN inherits the favorable long time properties of the Hamiltonian system, which in turn allows us to control the hidden states gradient with a hyperparameter of the Hamiltonian RNN architecture. This enables us to handle sequential learning problems with arbitrary sequence lengths, since for a range of values of this hyperparameter the gradient neither vanishes nor explodes. Additionally, we provide a heuristic for the optimal choice of the hyperparameter, which we use in our numerical simulations to illustrate that the Hamiltonian RNN is able to outperform other state-of-the-art RNNs without the need of computationally intensive hyperparameter optimization.