Transformer models are gauge invariant: A mathematical connection between AI and particle physics

TL;DR

This paper reveals that transformer models inherently possess gauge invariance properties similar to those in particle physics, highlighting a fundamental symmetry in AI architectures.

Contribution

It establishes a mathematical connection between transformer models and gauge invariance in particle physics, showing that transformers exhibit similar symmetry properties.

Findings

Transformers display gauge invariance-like properties.

Default transformer representations partially retain gauge invariance.

The work bridges concepts between AI and particle physics.

Abstract

In particle physics, the fundamental forces are subject to symmetries called gauge invariance. It is a redundancy in the mathematical description of any physical system. In this article I will demonstrate that the transformer architecture exhibits the same properties, and show that the default representation of transformers has partially, but not fully removed the gauge invariance.