Theoretical Constraints on the Expressive Power of $\mathsf{RoPE}$-based Tensor Attention Transformers

TL;DR

This paper provides a theoretical analysis of Tensor Attention and $ extsf{RoPE}$-based Tensor Attention, revealing their limitations in solving certain computational problems despite empirical successes, thus guiding future model development.

Contribution

It offers the first circuit complexity analysis of Tensor Attention and $ extsf{RoPE}$-based models, identifying their fundamental computational constraints.

Findings

Tensor Attention cannot solve fixed membership problems with polynomial precision.

$ extsf{RoPE}$-based Tensor Attention is limited in solving $(A_{F,r})^*$ closure problems.

Theoretical constraints are established under the assumption that $ extsf{TC}^0 eq extsf{NC}^1$.

Abstract

Tensor Attention extends traditional attention mechanisms by capturing high-order correlations across multiple modalities, addressing the limitations of classical matrix-based attention. Meanwhile, Rotary Position Embedding ($\mathsf{RoPE}$) has shown superior performance in encoding positional information in long-context scenarios, significantly enhancing transformer models' expressiveness. Despite these empirical successes, the theoretical limitations of these technologies remain underexplored. In this study, we analyze the circuit complexity of Tensor Attention and $\mathsf{RoPE}$-based Tensor Attention, showing that with polynomial precision, constant-depth layers, and linear or sublinear hidden dimension, they cannot solve fixed membership problems or $(A_{F,r})^*$ closure problems, under the assumption that $\mathsf{TC}^0 \neq \mathsf{NC}^1$. These findings highlight a gap between the empirical performance and theoretical constraints of Tensor Attention and $\mathsf{RoPE}$-based Tensor Attention Transformers, offering insights that could guide the development of more theoretically grounded approaches to Transformer model design and scaling.