The Geometry of Thought: Disclosing the Transformer as a Tropical Polynomial Circuit

TL;DR

This paper reveals that Transformer self-attention in the high-confidence limit functions as a tropical polynomial circuit, connecting neural computation with shortest-path algorithms and providing a new geometric perspective on reasoning.

Contribution

It demonstrates that in the high-confidence regime, Transformer attention operates as a tropical matrix product, linking neural networks to tropical algebra and dynamic programming.

Findings

Transformer attention converges to tropical matrix multiplication at high confidence.

The forward pass performs a shortest-path computation on token similarity graphs.

Provides a geometric interpretation of chain-of-thought reasoning in Transformers.

Abstract

We prove that the Transformer self-attention mechanism in the high-confidence regime ($\beta \to \infty$, where $\beta$ is an inverse temperature) operates in the tropical semiring (max-plus algebra). In particular, we show that taking the tropical limit of the softmax attention converts it into a tropical matrix product. This reveals that the Transformer's forward pass is effectively executing a dynamic programming recurrence (specifically, a Bellman-Ford path-finding update) on a latent graph defined by token similarities. Our theoretical result provides a new geometric perspective for chain-of-thought reasoning: it emerges from an inherent shortest-path (or longest-path) algorithm being carried out within the network's computation.