Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamics

TL;DR

This paper introduces a novel recurrent neural network architecture that enhances expressivity and stability for learning long-term dependencies by leveraging a Schur decomposition-based connectivity structure, surpassing traditional orthogonal RNNs.

Contribution

It proposes a new connectivity structure based on Schur decomposition that allows for more expressive recurrent matrices without sacrificing stability, unlike orthogonal RNNs.

Findings

Enhanced ability to learn long-term dependencies

Retains stability and training speed of orthogonal RNNs

Improved performance on sequence computation tasks

Abstract

A recent strategy to circumvent the exploding and vanishing gradient problem in RNNs, and to allow the stable propagation of signals over long time scales, is to constrain recurrent connectivity matrices to be orthogonal or unitary. This ensures eigenvalues with unit norm and thus stable dynamics and training. However this comes at the cost of reduced expressivity due to the limited variety of orthogonal transformations. We propose a novel connectivity structure based on the Schur decomposition and a splitting of the Schur form into normal and non-normal parts. This allows to parametrize matrices with unit-norm eigenspectra without orthogonality constraints on eigenbases. The resulting architecture ensures access to a larger space of spectrally constrained matrices, of which orthogonal matrices are a subset. This crucial difference retains the stability advantages and training speed of orthogonal RNNs while enhancing expressivity, especially on tasks that require computations over ongoing input sequences.