Lyapunov-Guided Representation of Recurrent Neural Network Performance

TL;DR

This paper introduces a novel approach that uses Lyapunov spectral analysis combined with autoencoders to relate RNN hyperparameters to performance, enabling early accuracy prediction and better understanding of RNN dynamics.

Contribution

It proposes a new method combining Lyapunov spectral analysis and autoencoders to interpret RNN performance and predict accuracy early in training.

Findings

AeLLE correlates Lyapunov spectrum with RNN accuracy

Latent representations generalize to new inputs

Early training representations predict final accuracy

Abstract

Recurrent Neural Networks (RNN) are ubiquitous computing systems for sequences and multivariate time series data. While several robust architectures of RNN are known, it is unclear how to relate RNN initialization, architecture, and other hyperparameters with accuracy for a given task. In this work, we propose to treat RNN as dynamical systems and to correlate hyperparameters with accuracy through Lyapunov spectral analysis, a methodology specifically designed for nonlinear dynamical systems. To address the fact that RNN features go beyond the existing Lyapunov spectral analysis, we propose to infer relevant features from the Lyapunov spectrum with an Autoencoder and an embedding of its latent representation (AeLLE). Our studies of various RNN architectures show that AeLLE successfully correlates RNN Lyapunov spectrum with accuracy. Furthermore, the latent representation learned by AeLLE is generalizable to novel inputs from the same task and is formed early in the process of RNN training. The latter property allows for the prediction of the accuracy to which RNN would converge when training is complete. We conclude that representation of RNN through Lyapunov spectrum along with AeLLE provides a novel method for organization and interpretation of variants of RNN architectures.