A Unified Geometric Field Theory Framework for Transformers: From Manifold Embeddings to Kernel Modulation

TL;DR

This paper introduces a unified geometric framework for Transformers, interpreting their components as kernel-modulated operators on manifolds, providing a new mathematical understanding of their core mechanisms.

Contribution

It develops a theoretical framework that unifies positional encoding, attention, and kernel operators within a geometric and field-theoretic perspective.

Findings

Maps discrete positions to continuous manifold functions

Provides a field-theoretic interpretation of Transformer layers

Lays groundwork for further theoretical analysis of Transformers

Abstract

The Transformer architecture has achieved tremendous success in natural language processing, computer vision, and scientific computing through its self-attention mechanism. However, its core components-positional encoding and attention mechanisms-have lacked a unified physical or mathematical interpretation. This paper proposes a structural theoretical framework that integrates positional encoding, kernel integral operators, and attention mechanisms for in-depth theoretical investigation. We map discrete positions (such as text token indices and image pixel coordinates) to spatial functions on continuous manifolds, enabling a field-theoretic interpretation of Transformer layers as kernel-modulated operators acting over embedded manifolds.