Recurrent Neural Networks as Weighted Language Recognizers

TL;DR

This paper analyzes the computational complexity of simple ReLU-activated RNNs used in NLP, revealing many problems are undecidable or computationally hard, thus emphasizing the need for heuristics in practical applications.

Contribution

It provides a formal complexity analysis of RNN recognition problems, showing undecidability and hardness results for common tasks.

Findings

Most recognition problems are undecidable.

Consistency makes the highest-weighted string problem decidable.

Limited string length reduces the problem to NP-complete.

Abstract

We investigate the computational complexity of various problems for simple recurrent neural networks (RNNs) as formal models for recognizing weighted languages. We focus on the single-layer, ReLU-activation, rational-weight RNNs with softmax, which are commonly used in natural language processing applications. We show that most problems for such RNNs are undecidable, including consistency, equivalence, minimization, and the determination of the highest-weighted string. However, for consistent RNNs the last problem becomes decidable, although the solution length can surpass all computable bounds. If additionally the string is limited to polynomial length, the problem becomes NP-complete and APX-hard. In summary, this shows that approximations and heuristic algorithms are necessary in practical applications of those RNNs.