Lower bounds for one-layer transformers that compute parity

TL;DR

This paper establishes fundamental lower bounds on the capacity of one-layer transformer models with rational post-processing to compute the parity function, highlighting limitations in their expressive power.

Contribution

It provides the first theoretical lower bounds showing that such transformers require linearly growing resources to compute parity, connecting rational approximation theory with neural network expressivity.

Findings

No self-attention layer with rational post-processing can sign-represent parity unless product of heads and degree grows linearly

Lower bounds extend to ReLU post-processing via rational approximation

Highlights fundamental limitations of one-layer transformers in computing parity

Abstract

This note shows that no self-attention layer post-processed by a rational function can sign-represent the parity function unless the product of the number of heads and the degree of the post-processing function grows linearly with the input length. Combining this lower bound with rational approximation of ReLU networks yields a margin-dependent extension for self-attention layers post-processed by ReLU networks.