Beyond Position: the emergence of wavelet-like properties in Transformers

TL;DR

This paper reveals that Transformer models with Rotary Position Embeddings spontaneously develop wavelet-like, multi-resolution processing capabilities during training, which help overcome positional encoding limitations and enhance model effectiveness.

Contribution

It uncovers the emergent wavelet-like properties in Transformers with RoPE, highlighting their unique multi-resolution processing and evolutionary development during training.

Findings

Attention heads evolve to implement multi-resolution processing.

Wavelet-like properties are unique to RoPE-based Transformers.

Emergence of these properties follows distinct training phases.

Abstract

This paper studies how Transformer models with Rotary Position Embeddings (RoPE) develop emergent, wavelet-like properties that compensate for the positional encoding's theoretical limitations. Through an analysis spanning model scales, architectures, and training checkpoints, we show that attention heads evolve to implement multi-resolution processing analogous to wavelet transforms. We demonstrate that this scale-invariant behavior is unique to RoPE, emerges through distinct evolutionary phases during training, and statistically adheres to the fundamental uncertainty principle. Our findings suggest that the effectiveness of modern Transformers stems from their remarkable ability to spontaneously develop optimal, multi-resolution decompositions to address inherent architectural constraints.