LieRE: Lie Rotational Positional Encodings

TL;DR

LieRE introduces a learnable, high-dimensional rotational positional encoding for transformers, improving their ability to model complex spatial structures in vision tasks while maintaining efficiency.

Contribution

We propose LieRE, a novel generalization of RoPE that learns dense skew-symmetric matrices to encode positional information in high-dimensional spaces.

Findings

LieRE outperforms traditional RoPE in 2D and 3D vision tasks.

LieRE generalizes well to higher resolutions.

It maintains computational efficiency.

Abstract

Transformer architectures rely on position encodings to model the spatial structure of input data. Rotary Position Encoding (RoPE) is a widely used method in language models that encodes relative positions through fixed, block-diagonal, rotation matrices applied to key-query interactions. We hypothesize that this inductive bias limits their RoPE's effectiveness for modalities with high dimensional structure. Lie Relative Encodings (LieRE) introduce a principled generalization of RoPE, aimed at increasing the representational capacity of positional encodings in transformers. Instead of fixed 2D rotations, LieRE learns dense skew-symmetric matrices (Lie algebra elements), which are then differentiable mapped to form high-dimensional rotation matrices (Lie group elements). This results in richer, learnable, and continuous, encodings of both relative and absolute positional information. We demonstrate the effectiveness of LieRE on 2D and 3D vision tasks, showing that it generalizes well to higher input resolutions while maintaining computational efficiency. The code and checkpoints are publicly available at https://github.com/StanfordMIMI/LieRE.