JoFormer (Journey-based Transformer): Theory and Empirical Analysis on the Tiny Shakespeare Dataset

TL;DR

JoFormer introduces a journey-based Transformer architecture that leverages non-commutative algebra for positional encoding, outperforming standard models like RoFormer on the Tiny Shakespeare dataset with lower perplexity and faster convergence.

Contribution

The paper presents a novel journey-based Transformer architecture, JoFormer, grounded in non-commutative algebra, extending relative position representations and subsuming existing methods like rotary transformations.

Findings

JoFormer achieves lower perplexity than RoFormer on Tiny Shakespeare.

JoFormer demonstrates faster convergence in language modeling tasks.

The approach offers a more expressive, principled way to incorporate positional information.

Abstract

Transformers have demonstrated remarkable success in sequence modeling, yet effectively incorporating positional information remains a challenging and active area of research. In this paper, we introduce JoFormer, a journey-based Transformer architecture grounded in a recently proposed non-commutative algebra for composing transformations across positions. JoFormer represents relative positions through learnable directional transforms that are sequentially composed along the input, thereby extending and generalizing existing approaches based on relative position representations. We derive the JoFormer attention mechanism from first principles and show that it subsumes standard methods such as rotary transformations as special cases. To evaluate its effectiveness, we compare JoFormer to the RoFormer baseline on the Tiny Shakespeare character-level language modeling task. Our results demonstrate that JoFormer consistently achieves lower perplexity and faster convergence, highlighting the advantages of its more expressive, journey-based treatment of position. Notably, the per-token JoFormer is still a primitive, conceptual variant with layer-independent angles, yet it already demonstrates strong performance-underscoring its promise as a proof of concept for more expressive architectures. We conclude by discussing how JoFormer offers a principled approach to integrating positional structure into Transformer architectures. The code used in this work is available at https://github.com/mahesh-godavarti/joformer.