Space filling positionality and the Spiroformer

TL;DR

The paper introduces the Spiroformer, a transformer model that employs space-filling curves to handle data on geometric domains like manifolds, addressing the challenge of undefined global order.

Contribution

It proposes a novel attention mechanism using space-filling curves for geometric data, exemplified by the Spiroformer on the 2-sphere.

Findings

Demonstrates the effectiveness of space-filling attention heads on spherical data

Introduces a new transformer architecture for geometric domains

Shows potential for generalizing transformers to manifolds

Abstract

Transformers excel when dealing with sequential data. Generalizing transformer models to geometric domains, such as manifolds, we encounter the problem of not having a well-defined global order. We propose a solution with attention heads following a space-filling curve. As a first experimental example, we present the Spiroformer, a transformer that follows a polar spiral on the $2$-sphere.