Toward Manifest Relationality in Transformers via Symmetry Reduction

TL;DR

This paper introduces a symmetry reduction framework for Transformer models that reformulates their components in terms of invariant relational quantities, reducing redundancy and providing a geometric perspective.

Contribution

It presents a novel symmetry reduction approach that reformulates Transformer representations and attention mechanisms using invariant relational structures, enhancing efficiency and interpretability.

Findings

Reduces parameter redundancy in Transformers.

Provides a geometric framework for analyzing optimization.

Operates directly on relational structures.

Abstract

Transformer models contain substantial internal redundancy arising from coordinate-dependent representations and continuous symmetries, in model space and in head space, respectively. While recent approaches address this by explicitly breaking symmetry, we propose a complementary framework based on symmetry reduction. We reformulate representations, attention mechanisms, and optimization dynamics in terms of invariant relational quantities, eliminating redundant degrees of freedom by construction. This perspective yields architectures that operate directly on relational structures, providing a principled geometric framework for reducing parameter redundancy and analyzing optimization.