A Survey of Neural Networks and Formal Languages

TL;DR

This survey explores how different neural network architectures relate to formal languages, focusing on their ability to recognize, generate, and learn languages within the Chomsky hierarchy.

Contribution

It provides a comprehensive overview of the capabilities of neural networks in modeling formal languages, highlighting current understanding and gaps.

Findings

Neural networks can recognize certain regular and context-free languages.

Representation of complex languages remains challenging for current architectures.

Learning from samples enables neural networks to approximate language structures.

Abstract

This report is a survey of the relationships between various state-of-the-art neural network architectures and formal languages as, for example, structured by the Chomsky Language Hierarchy. Of particular interest are the abilities of a neural architecture to represent, recognize and generate words from a specific language by learning from samples of the language.