Recurrent Neural Language Models as Probabilistic Finite-state Automata

TL;DR

This paper characterizes the class of probability distributions over strings that recurrent neural network language models can represent, showing their equivalence to a subclass of probabilistic finite-state automata and analyzing their complexity.

Contribution

It establishes a formal connection between RNN language models and probabilistic finite-state automata, providing insights into their representational capabilities and limitations.

Findings

Simple RNNs are equivalent to a subclass of probabilistic finite-state automata.

RNNs require at least Ω(N|Σ|) neurons to represent arbitrary deterministic finite-state models.

The work offers a formal framework to understand what distributions RNN language models can express.

Abstract

Studying language models (LMs) in terms of well-understood formalisms allows us to precisely characterize their abilities and limitations. Previous work has investigated the representational capacity of recurrent neural network (RNN) LMs in terms of their capacity to recognize unweighted formal languages. However, LMs do not describe unweighted formal languages -- rather, they define \emph{probability distributions} over strings. In this work, we study what classes of such probability distributions RNN LMs can represent, which allows us to make more direct statements about their capabilities. We show that simple RNNs are equivalent to a subclass of probabilistic finite-state automata, and can thus model a strict subset of probability distributions expressible by finite-state models. Furthermore, we study the space complexity of representing finite-state LMs with RNNs. We show that, to represent an arbitrary deterministic finite-state LM with $N$ states over an alphabet $\alphabet$, an RNN requires $\Omega\left(N |\Sigma|\right)$ neurons. These results present a first step towards characterizing the classes of distributions RNN LMs can represent and thus help us understand their capabilities and limitations.