Approximation and Estimation Ability of Transformers for Sequence-to-Sequence Functions with Infinite Dimensional Input

TL;DR

This paper provides a theoretical analysis of Transformers' ability to approximate and estimate sequence-to-sequence functions with infinite-dimensional inputs, explaining their success in high-dimensional tasks.

Contribution

It offers the first theoretical insights into how Transformers handle infinite-dimensional inputs and adapt to varying smoothness, avoiding the curse of dimensionality.

Findings

Transformers can avoid the curse of dimensionality with anisotropic smoothness.

They can dynamically estimate feature importance based on input.

Transformers achieve optimal convergence rates similar to fixed smoothness cases.

Abstract

Despite the great success of Transformer networks in various applications such as natural language processing and computer vision, their theoretical aspects are not well understood. In this paper, we study the approximation and estimation ability of Transformers as sequence-to-sequence functions with infinite dimensional inputs. Although inputs and outputs are both infinite dimensional, we show that when the target function has anisotropic smoothness, Transformers can avoid the curse of dimensionality due to their feature extraction ability and parameter sharing property. In addition, we show that even if the smoothness changes depending on each input, Transformers can estimate the importance of features for each input and extract important features dynamically. Then, we proved that Transformers achieve similar convergence rate as in the case of the fixed smoothness. Our theoretical results support the practical success of Transformers for high dimensional data.