Parity, Sensitivity, and Transformers

TL;DR

This paper investigates the capabilities of transformers to compute the PARITY task, establishing that at least two layers are needed and providing a practical four-layer construction that removes previous impractical assumptions.

Contribution

It proves that one-layer transformers cannot compute PARITY and introduces a new four-layer transformer architecture that overcomes previous limitations.

Findings

One-layer transformers grow slower in sensitivity than PARITY.

Two-layer transformers are necessary to compute PARITY.

A practical four-layer transformer can compute PARITY without impractical assumptions.

Abstract

Understanding what neural architectures can and cannot compute is a central challenge in the theory of AI. One of the fundamental problems in this context is the PARITY task, which asks whether the number of 1s in a binary input sequence is even or odd. PARITY is one of the central tasks studied in the theory of computation, yet it remains surprisingly unclear under which conditions transformers can or cannot solve it. In this paper, we show that the minimal number of layers a transformer needs to compute PARITY is two. In particular, we solve the open problem asking whether a one-layer transformer can compute PARITY. We answer it negatively by showing that average sensitivity of a one-layer transformer grows slower than that of PARITY. Furthermore, we show a new construction for transformer that computes PARITY, which improves on the existing constructions by removing a number of impractical assumptions. In particular, the existing transformers for PARITY rely on such impractical assumptions as length-dependent positional encoding, hardmax, layernorm without a regularisation parameter, or incompatibility with causal masking. We show that these assumptions can be removed, at the cost of increasing the number of layers from two to four. Specifically, we show that PARITY can be computed by a four-layer transformer using softmax attention, length-independent and polynomially bounded positional encoding, no layernorm, and compatible with both causal and non-causal masking.