Average-Hard Attention Transformers are Constant-Depth Uniform Threshold Circuits

TL;DR

This paper demonstrates that average-hard attention transformers can be simulated by uniform constant-depth threshold circuits, extending previous results that linked transformers with non-uniform circuits, thus providing a deeper understanding of their computational power.

Contribution

It extends prior work by proving that average-hard attention transformers are equivalent to uniform TC0 circuits, highlighting their computational robustness and uniformity.

Findings

Transformers recognize languages in TC0 class.

Uniform TC0 circuits can simulate average-hard attention transformers.

The results unify transformer models with classical circuit complexity classes.

Abstract

Transformers have emerged as a widely used neural network model for various natural language processing tasks. Previous research explored their relationship with constant-depth threshold circuits, making two assumptions: average-hard attention and logarithmic precision for internal computations relative to input length. Merrill et al. (2022) prove that average-hard attention transformers recognize languages that fall within the complexity class TC0, denoting the set of languages that can be recognized by constant-depth polynomial-size threshold circuits. Likewise, Merrill and Sabharwal (2023) show that log-precision transformers recognize languages within the class of uniform TC0. This shows that both transformer models can be simulated by constant-depth threshold circuits, with the latter being more robust due to generating a uniform circuit family. Our paper shows that the first result can be extended to yield uniform circuits as well.