A completely uniform transformer for parity

TL;DR

This paper introduces a 3-layer uniform transformer capable of recognizing the parity language without input-length-dependent parameters or positional encoding, advancing the understanding of transformer capabilities.

Contribution

It presents a novel 3-layer transformer architecture that recognizes parity with fixed parameters and no positional encoding, improving upon previous length-dependent models.

Findings

Recognizes parity with a 3-layer uniform transformer

Eliminates the need for input-length-dependent positional encoding

Simplifies transformer architecture for specific language recognition

Abstract

We construct a 3-layer constant-dimension transformer, recognizing the parity language, where neither parameter matrices nor the positional encoding depend on the input length. This improves upon a construction of Chiang and Cholak who use a positional encoding, depending on the input length (but their construction has 2 layers).