DizzyRNN: Reparameterizing Recurrent Neural Networks for Norm-Preserving Backpropagation

TL;DR

DizzyRNN introduces a reparameterization technique for recurrent neural networks that preserves norms during backpropagation, effectively addressing vanishing and exploding gradients for better learning of long-term dependencies.

Contribution

It proposes a novel reparameterization using Givens rotations and an absolute value non-linearity to maintain norm preservation, reducing parameters and improving performance.

Findings

Outperforms standard RNNs with orthogonal init and LSTMs on the copy problem.

Maintains same complexity as standard RNNs while reducing parameters.

Proven to be norm-preserving for better gradient flow.

Abstract

The vanishing and exploding gradient problems are well-studied obstacles that make it difficult for recurrent neural networks to learn long-term time dependencies. We propose a reparameterization of standard recurrent neural networks to update linear transformations in a provably norm-preserving way through Givens rotations. Additionally, we use the absolute value function as an element-wise non-linearity to preserve the norm of backpropagated signals over the entire network. We show that this reparameterization reduces the number of parameters and maintains the same algorithmic complexity as a standard recurrent neural network, while outperforming standard recurrent neural networks with orthogonal initializations and Long Short-Term Memory networks on the copy problem.