# LSTM Networks Can Perform Dynamic Counting

## TL;DR

This paper demonstrates that small LSTM networks can perform dynamic counting and recognize certain formal languages, like Dyck-$1$ and shuffled variants, but struggle with more complex languages requiring stack mechanisms.

## Contribution

It is the first study to analyze neural networks' ability to recognize shuffle languages and shows LSTMs can emulate simple counter machines for language recognition.

## Key findings

- LSTMs can recognize Dyck-$1$ and shuffled Dyck-$1$ languages.
- A single LSTM unit suffices for Dyck-$1$ recognition.
- LSTMs fail to learn Dyck-$2$, which needs stack-like memory.

## Abstract

In this paper, we systematically assess the ability of standard recurrent networks to perform dynamic counting and to encode hierarchical representations. All the neural models in our experiments are designed to be small-sized networks both to prevent them from memorizing the training sets and to visualize and interpret their behaviour at test time. Our results demonstrate that the Long Short-Term Memory (LSTM) networks can learn to recognize the well-balanced parenthesis language (Dyck-$1$) and the shuffles of multiple Dyck-$1$ languages, each defined over different parenthesis-pairs, by emulating simple real-time $k$-counter machines. To the best of our knowledge, this work is the first study to introduce the shuffle languages to analyze the computational power of neural networks. We also show that a single-layer LSTM with only one hidden unit is practically sufficient for recognizing the Dyck-$1$ language. However, none of our recurrent networks was able to yield a good performance on the Dyck-$2$ language learning task, which requires a model to have a stack-like mechanism for recognition.