# Sequential Neural Networks as Automata

## TL;DR

This paper explores the computational capabilities of neural networks by relating them to automata theory, providing a theoretical framework to understand their memory and language recognition abilities.

## Contribution

It introduces a formal definition of neural network acceptance of languages, characterizes various networks' language classes, and links neural models to automata and formal language hierarchies.

## Key findings

- LSTMs act like counter machines
- Convolutional networks relate to subregular hierarchy
- Provides a theoretical framework for neural network interpretation

## Abstract

This work attempts to explain the types of computation that neural networks can perform by relating them to automata. We first define what it means for a real-time network with bounded precision to accept a language. A measure of network memory follows from this definition. We then characterize the classes of languages acceptable by various recurrent networks, attention, and convolutional networks. We find that LSTMs function like counter machines and relate convolutional networks to the subregular hierarchy. Overall, this work attempts to increase our understanding and ability to interpret neural networks through the lens of theory. These theoretical insights help explain neural computation, as well as the relationship between neural networks and natural language grammar.