Formal Language Theory Meets Modern NLP

TL;DR

This paper explores the intersection of formal language theory and modern NLP, highlighting how formal analysis of neural network models can deepen understanding of language processing in AI.

Contribution

It provides a background overview connecting formal language concepts with recent neural network analyses in NLP, emphasizing their relevance and application.

Findings

Formal languages underpin modern NLP models.

Recent work analyzes neural networks using formal language theory.

Bridging formal language theory and deep learning enhances understanding of NLP models.

Abstract

NLP is deeply intertwined with the formal study of language, both conceptually and historically. Arguably, this connection goes all the way back to Chomsky's Syntactic Structures in 1957. It also still holds true today, with a strand of recent works building formal analysis of modern neural networks methods in terms of formal languages. In this document, I aim to explain background about formal languages as they relate to this recent work. I will by necessity ignore large parts of the rich history of this field, instead focusing on concepts connecting to modern deep learning-based NLP.