Transformers Can Represent $n$-gram Language Models

TL;DR

This paper demonstrates that transformer language models with hard or sparse attention can exactly represent any n-gram model, providing insights into their capacity to model probability distributions over strings.

Contribution

It establishes that transformer LMs with specific attention mechanisms can exactly encode n-gram models, advancing understanding of their probabilistic representational capabilities.

Findings

Transformer LMs with hard or sparse attention can represent any n-gram LM.

Provides a lower bound on the probabilistic capacity of transformer LMs.

First step towards understanding how transformers model string probability distributions.

Abstract

Existing work has analyzed the representational capacity of the transformer architecture by means of formal models of computation. However, the focus so far has been on analyzing the architecture in terms of language \emph{acceptance}. We contend that this is an ill-suited problem in the study of \emph{language models} (LMs), which are definitionally \emph{probability distributions} over strings. In this paper, we focus on the relationship between transformer LMs and $n$-gram LMs, a simple and historically relevant class of language models. We show that transformer LMs using the hard or sparse attention mechanisms can exactly represent any $n$-gram LM, giving us a concrete lower bound on their probabilistic representational capacity. This provides a first step towards understanding the mechanisms that transformer LMs can use to represent probability distributions over strings.