How Chaotic Are Recurrent Neural Networks?

TL;DR

This paper empirically investigates the chaotic behavior of recurrent neural networks, finding that vanilla and LSTM RNNs do not exhibit chaos during training in real-world tasks, challenging previous assumptions.

Contribution

It provides the first systematic empirical analysis showing RNNs do not display chaos in practical applications, shifting focus for future research.

Findings

RNNs do not exhibit chaos during training in text generation tasks

Vanilla and LSTM RNNs are stable in real-world applications

Challenges prior beliefs about chaos in RNN dynamics

Abstract

Recurrent neural networks (RNNs) are non-linear dynamic systems. Previous work believes that RNN may suffer from the phenomenon of chaos, where the system is sensitive to initial states and unpredictable in the long run. In this paper, however, we perform a systematic empirical analysis, showing that a vanilla or long short term memory (LSTM) RNN does not exhibit chaotic behavior along the training process in real applications such as text generation. Our findings suggest that future work in this direction should address the other side of non-linear dynamics for RNN.