Comparison of different Unique hard attention transformer models by the formal languages they can recognize

TL;DR

This paper surveys the recognition capabilities of various unique hard attention transformer models, analyzing their formal language recognition power based on different configurations and theoretical bounds.

Contribution

It systematically compares UHAT models with different attention mechanisms and establishes theoretical bounds on their language recognition capabilities.

Findings

Different UHAT configurations have varying recognition powers.

The models are characterized by bounds in logic and circuit complexity.

The survey clarifies the relationships between model types and their formal language classes.

Abstract

This note is a survey of various results on the capabilities of unique hard attention transformers encoders (UHATs) to recognize formal languages. We distinguish between masked vs. non-masked, finite vs. infinite image and general vs. bilinear attention score functions. We recall some relations between these models, as well as a lower bound in terms of first-order logic and an upper bound in terms of circuit complexity.