TL;DR
Tropical Attention introduces a geometry-grounded attention mechanism that enhances neural reasoning models' robustness, interpretability, and ability to handle complex combinatorial problems beyond PTIME, with improved generalization and efficiency.
Contribution
The paper presents Tropical Attention, a novel attention mechanism based on tropical geometry, enabling universal approximation of tropical circuits and extending neural reasoning to NP-hard problems.
Findings
Stronger out-of-distribution generalization in length and value.
High robustness against perturbative noise.
Faster inference with fewer parameters.
Abstract
Can algebraic geometry enhance the sharpness, robustness, and interpretability of modern neural reasoning models by equipping them with a mathematically grounded inductive bias? To answer this, we introduce Tropical Attention, an attention mechanism grounded in tropical geometry that lifts the attention kernel into tropical projective space, where reasoning is piecewise-linear and 1-Lipschitz, thus preserving the polyhedral decision structure inherent to combinatorial reasoning. We prove that Multi-Head Tropical Attention (MHTA) stacks universally approximate tropical circuits and realize tropical transitive closure through composition, achieving polynomial resource bounds without invoking recurrent mechanisms. These guarantees explain why the induced polyhedral decision boundaries remain sharp and scale-invariant, rather than smoothed by Softmax. Empirically, we show that Tropical…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
Taxonomy
MethodsAttention Is All You Need · Softmax
