Spherical Position Encoding for Transformers

TL;DR

This paper introduces a spherical position encoding mechanism for transformers, called geotokens, which effectively captures geographical coordinates and their relative distances, improving modeling of spatial data.

Contribution

It proposes a novel spherical position encoding method based on RoPE, tailored for geospatial data, extending transformer capabilities beyond sequential language tasks.

Findings

Effective encoding of geographical coordinates in transformers

Maintains proportional distances on spherical surfaces

Enhances spatial data modeling in transformer architectures

Abstract

Position encoding is the primary mechanism which induces notion of sequential order for input tokens in transformer architectures. Even though this formulation in the original transformer paper has yielded plausible performance for general purpose language understanding and generation, several new frameworks such as Rotary Position Embedding (RoPE) are proposed for further enhancement. In this paper, we introduce the notion of "geotokens" which are input elements for transformer architectures, each representing an information related to a geological location. Unlike the natural language the sequential position is not important for the model but the geographical coordinates are. In order to induce the concept of relative position for such a setting and maintain the proportion between the physical distance and distance on embedding space, we formulate a position encoding mechanism based on RoPE architecture which is adjusted for spherical coordinates.