Theoretical Analysis of Positional Encodings in Transformer Models: Impact on Expressiveness and Generalization

TL;DR

This paper provides a theoretical analysis of different positional encoding methods in transformers, examining their impact on expressiveness, generalization, and extrapolation, and introduces new orthogonal function-based encodings with promising results.

Contribution

It offers a unified theoretical framework for understanding positional encodings, proposes novel orthogonal encodings, and evaluates their effectiveness in generalization and extrapolation tasks.

Findings

Orthogonal function-based encodings outperform sinusoidal encodings in generalization.

Theoretical bounds relate encoding choice to model expressiveness and extrapolation.

Experimental results support the advantages of wavelet and Legendre polynomial encodings.

Abstract

Positional encodings are a core part of transformer-based models, enabling processing of sequential data without recurrence. This paper presents a theoretical framework to analyze how various positional encoding methods, including sinusoidal, learned, relative, and bias-based methods like Attention with Linear Biases (ALiBi), impact a transformer's expressiveness, generalization ability, and extrapolation to longer sequences. Expressiveness is defined via function approximation, generalization bounds are established using Rademacher complexity, and new encoding methods based on orthogonal functions, such as wavelets and Legendre polynomials, are proposed. The extrapolation capacity of existing and proposed encodings is analyzed, extending ALiBi's biasing approach to a unified theoretical context. Experimental evaluation on synthetic sequence-to-sequence tasks shows that orthogonal transform-based encodings outperform traditional sinusoidal encodings in generalization and extrapolation. This work addresses a critical gap in transformer theory, providing insights for design choices in natural language processing, computer vision, and other transformer applications.