Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling

TL;DR

This paper provides a theoretical and experimental analysis of Transformer models, revealing how their components influence expressive power and offering insights for designing improved architectures.

Contribution

It systematically studies the approximation capabilities of Transformers for complex sequence modeling and clarifies the roles of key components and parameters.

Findings

Transformer components significantly impact expressive power

Explicit approximation rates are established for different architectures

Experimental validation supports theoretical insights

Abstract

We conduct a systematic study of the approximation properties of Transformer for sequence modeling with long, sparse and complicated memory. We investigate the mechanisms through which different components of Transformer, such as the dot-product self-attention, positional encoding and feed-forward layer, affect its expressive power, and we study their combined effects through establishing explicit approximation rates. Our study reveals the roles of critical parameters in the Transformer, such as the number of layers and the number of attention heads. These theoretical insights are validated experimentally and offer natural suggestions for alternative architectures.