On the Expressive Power of Floating-Point Transformers

TL;DR

This paper investigates the expressive power of floating-point transformers, revealing their capabilities and limitations in representing permutation-equivariant and non-equivariant functions under finite-precision arithmetic.

Contribution

It extends theoretical understanding by analyzing how floating-point constraints affect the representability of functions in transformers, including the impact of positional encoding.

Findings

Floating-point transformers can represent some non-permutation-equivariant functions without positional encoding.

They can represent all permutation-equivariant functions when sequence length is bounded.

Large sequence lengths limit the representability of permutation-equivariant functions.

Abstract

The study on the expressive power of transformers shows that transformers are permutation equivariant, and they can approximate all permutation-equivariant continuous functions on a compact domain. However, these results are derived under real parameters and exact operations, while real implementations on computers can only use a finite set of numbers and inexact machine operations with round-off errors. In this work, we investigate the representability of floating-point transformers that use floating-point parameters and floating-point operations. Unlike existing results under exact operations, we first show that floating-point transformers can represent a class of non-permutation-equivariant functions even without positional encoding. Furthermore, we prove that floating-point transformers can represent all permutation-equivariant functions when the sequence length is bounded, but they cannot when the sequence length is large. We also found the minimal equivariance structure in floating-point transformers, and show that all non-trivial additive positional encoding can harm the representability of floating-point transformers.