Counting in Language with RNNs

TL;DR

This paper investigates why LSTM models outperform GRUs in language tasks, attributing the difference to their ability to perform counting based on cell states, especially in simplified language models.

Contribution

It provides a theoretical analysis demonstrating how LSTMs perform counting through cell states, explaining their superior performance over GRUs in language modeling.

Findings

LSTMs can perform counting in simplified language models

GRUs are less capable of counting due to their structure

Counting ability correlates with language modeling performance

Abstract

In this paper we examine a possible reason for the LSTM outperforming the GRU on language modeling and more specifically machine translation. We hypothesize that this has to do with counting. This is a consistent theme across the literature of long term dependence, counting, and language modeling for RNNs. Using the simplified forms of language -- Context-Free and Context-Sensitive Languages -- we show how exactly the LSTM performs its counting based on their cell states during inference and why the GRU cannot perform as well.