Subregular Complexity and Deep Learning

TL;DR

This paper uses formal language theory and grammatical inference to evaluate how different RNN architectures learn long-term dependencies, revealing that LSTMs do not always outperform simpler RNNs in complex sequence tasks.

Contribution

It demonstrates that LSTMs may not be superior to simple RNNs in learning certain long-term dependencies, challenging common assumptions in deep learning.

Findings

LSTMs performed worse on long-term dependency tasks than on local ones.

Simple RNNs outperformed LSTMs on complex long-term dependency tasks.

Results question the assumption that LSTMs are always better at learning long-term dependencies.

Abstract

This paper argues that the judicial use of formal language theory and grammatical inference are invaluable tools in understanding how deep neural networks can and cannot represent and learn long-term dependencies in temporal sequences. Learning experiments were conducted with two types of Recurrent Neural Networks (RNNs) on six formal languages drawn from the Strictly Local (SL) and Strictly Piecewise (SP) classes. The networks were Simple RNNs (s-RNNs) and Long Short-Term Memory RNNs (LSTMs) of varying sizes. The SL and SP classes are among the simplest in a mathematically well-understood hierarchy of subregular classes. They encode local and long-term dependencies, respectively. The grammatical inference algorithm Regular Positive and Negative Inference (RPNI) provided a baseline. According to earlier research, the LSTM architecture should be capable of learning long-term dependencies and should outperform s-RNNs. The results of these experiments challenge this narrative. First, the LSTMs' performance was generally worse in the SP experiments than in the SL ones. Second, the s-RNNs out-performed the LSTMs on the most complex SP experiment and performed comparably to them on the others.