A Circular Argument : Does RoPE need to be Equivariant for Vision?

TL;DR

This paper investigates the importance of equivariance in Rotary Positional Encodings for vision, showing that non-equivariant variants can perform equally well or better, challenging common assumptions.

Contribution

It introduces Spherical RoPE, a non-equivariant positional encoding, and demonstrates its effectiveness, questioning the necessity of equivariance in vision tasks.

Findings

Spherical RoPE matches or outperforms equivariant variants in vision tasks.

Mathematical analysis shows RoPE's generality for equivariant embeddings.

Non-equivariant encodings can be faster and more flexible for vision applications.

Abstract

Rotary Positional Encodings (RoPE) have emerged as a highly effective technique for one-dimensional sequences in Natural Language Processing spurring recent progress towards generalizing RoPE to higher-dimensional data such as images and videos. The success of RoPE has been thought to be due to its positional equivariance, i.e. its status as a relative positional encoding. In this paper, we mathematically show RoPE to be one of the most general solutions for equivariant positional embedding in one-dimensional data. Moreover, we show Mixed RoPE to be the analogously general solution for M-dimensional data, if we require commutative generators -- a property necessary for RoPE's equivariance. However, we question whether strict equivariance plays a large role in RoPE's performance. We propose Spherical RoPE, a method analogous to Mixed RoPE, but assumes non-commutative generators. Empirically, we find Spherical RoPE to have the equivalent or better learning behavior compared to its equivariant analogues. This suggests that relative positional embeddings are not as important as is commonly believed, at least within computer vision. We expect this discovery to facilitate future work in positional encodings for vision that can be faster and generalize better by removing the preconception that they must be relative.