A Unified Framework of Online Learning Algorithms for Training Recurrent Neural Networks

TL;DR

This paper introduces a comprehensive framework categorizing recent online learning algorithms for training recurrent neural networks, revealing their underlying connections and providing insights into their effectiveness.

Contribution

It offers a unified classification scheme for online RNN training algorithms and introduces new mathematical intuitions for understanding their success.

Findings

Algorithms cluster based on proposed criteria

Performance does not solely depend on gradient alignment

Better comparison metrics are needed for stochastic algorithms

Abstract

We present a framework for compactly summarizing many recent results in efficient and/or biologically plausible online training of recurrent neural networks (RNN). The framework organizes algorithms according to several criteria: (a) past vs. future facing, (b) tensor structure, (c) stochastic vs. deterministic, and (d) closed form vs. numerical. These axes reveal latent conceptual connections among several recent advances in online learning. Furthermore, we provide novel mathematical intuitions for their degree of success. Testing various algorithms on two synthetic tasks shows that performances cluster according to our criteria. Although a similar clustering is also observed for gradient alignment, alignment with exact methods does not alone explain ultimate performance, especially for stochastic algorithms. This suggests the need for better comparison metrics.