adaQN: An Adaptive Quasi-Newton Algorithm for Training RNNs

TL;DR

adaQN is a new stochastic quasi-Newton method for training RNNs that balances low per-iteration cost with effective curvature approximation, improving training efficiency on language modeling tasks.

Contribution

The paper introduces adaQN, a stochastic quasi-Newton algorithm with a novel L-BFGS scaling scheme for efficient RNN training.

Findings

adaQN is competitive with popular RNN training algorithms.

The method effectively balances curvature information and computational cost.

Numerical experiments demonstrate its practical effectiveness on language modeling tasks.

Abstract

Recurrent Neural Networks (RNNs) are powerful models that achieve exceptional performance on several pattern recognition problems. However, the training of RNNs is a computationally difficult task owing to the well-known "vanishing/exploding" gradient problem. Algorithms proposed for training RNNs either exploit no (or limited) curvature information and have cheap per-iteration complexity, or attempt to gain significant curvature information at the cost of increased per-iteration cost. The former set includes diagonally-scaled first-order methods such as ADAGRAD and ADAM, while the latter consists of second-order algorithms like Hessian-Free Newton and K-FAC. In this paper, we present adaQN, a stochastic quasi-Newton algorithm for training RNNs. Our approach retains a low per-iteration cost while allowing for non-diagonal scaling through a stochastic L-BFGS updating scheme. The method uses a novel L-BFGS scaling initialization scheme and is judicious in storing and retaining L-BFGS curvature pairs. We present numerical experiments on two language modeling tasks and show that adaQN is competitive with popular RNN training algorithms.