Bifurcations and loss jumps in RNN training

TL;DR

This paper investigates bifurcations in ReLU-based RNNs, mathematically links them to loss jumps during training, and introduces an exact algorithm for detecting fixed points and cycles, enhancing understanding of RNN dynamics and training behavior.

Contribution

It provides a mathematical proof connecting bifurcations to loss gradients and introduces a novel, exact heuristic algorithm for detecting fixed points and cycles in ReLU RNNs, improving analysis tools.

Findings

Bifurcations are linked to loss jumps in RNN training.

The new algorithm accurately finds fixed points and cycles in ReLU RNNs.

Generalized teacher forcing avoids certain bifurcations during training.

Abstract

Recurrent neural networks (RNNs) are popular machine learning tools for modeling and forecasting sequential data and for inferring dynamical systems (DS) from observed time series. Concepts from DS theory (DST) have variously been used to further our understanding of both, how trained RNNs solve complex tasks, and the training process itself. Bifurcations are particularly important phenomena in DS, including RNNs, that refer to topological (qualitative) changes in a system's dynamical behavior as one or more of its parameters are varied. Knowing the bifurcation structure of an RNN will thus allow to deduce many of its computational and dynamical properties, like its sensitivity to parameter variations or its behavior during training. In particular, bifurcations may account for sudden loss jumps observed in RNN training that could severely impede the training process. Here we first mathematically prove for a particular class of ReLU-based RNNs that certain bifurcations are indeed associated with loss gradients tending toward infinity or zero. We then introduce a novel heuristic algorithm for detecting all fixed points and k-cycles in ReLU-based RNNs and their existence and stability regions, hence bifurcation manifolds in parameter space. In contrast to previous numerical algorithms for finding fixed points and common continuation methods, our algorithm provides exact results and returns fixed points and cycles up to high orders with surprisingly good scaling behavior. We exemplify the algorithm on the analysis of the training process of RNNs, and find that the recently introduced technique of generalized teacher forcing completely avoids certain types of bifurcations in training. Thus, besides facilitating the DST analysis of trained RNNs, our algorithm provides a powerful instrument for analyzing the training process itself.