RiemannFormer: A Framework for Attention in Curved Spaces

TL;DR

RiemannFormer introduces a geometric framework for attention in transformers using Riemannian geometry, enhancing performance by incorporating local structure and reducing parameters.

Contribution

It provides a novel geometric interpretation of attention mechanisms in transformers, utilizing metric tensors and tangent spaces, with parameter reduction techniques.

Findings

Significant performance improvements over baseline models.

Effective incorporation of local geometric structure.

Potential for enhanced transformer efficiency and accuracy.

Abstract

This research endeavors to offer insights into unlocking the further potential of transformer-based architectures. One of the primary motivations is to offer a geometric interpretation for the attention mechanism in transformers. In our framework, the attention mainly involves metric tensors, tangent spaces, inner product, and how they relate to each other. These quantities and structures at discrete positions are intricately interconnected via the parallel transport of tangent vectors. To make the learning process more efficient, we reduce the number of parameters through ingenious predefined configurations. Moreover, we introduce an explicit mechanism to highlight a neighborhood by attenuating the remote values, given that transformers inherently neglect local inductive bias. Experimental results demonstrate that our modules deliver significant performance improvements relative to the baseline. More evaluation experiments on visual and large language models will be launched successively.