Neural Isomorphic Fields: A Transformer-based Algebraic Numerical Embedding

TL;DR

This paper introduces Neural Isomorphic Fields, a transformer-based approach that creates fixed-length number embeddings preserving algebraic operations, improving numerical stability and algebraic property testing in neural models.

Contribution

It presents the first fixed-length neural embeddings that preserve algebraic structures like fields, enabling stable and algebraically consistent neural computations.

Findings

Addition achieves over 95% accuracy on algebraic tests

Multiplication accuracy ranges from 53% to 73%

Embeddings improve numerical stability in neural networks

Abstract

Neural network models often face challenges when processing very small or very large numbers due to issues such as overflow, underflow, and unstable output variations. To mitigate these problems, we propose using embedding vectors for numbers instead of directly using their raw values. These embeddings aim to retain essential algebraic properties while preventing numerical instabilities. In this paper, we introduce, for the first time, a fixed-length number embedding vector that preserves algebraic operations, including addition, multiplication, and comparison, within the field of rational numbers. We propose a novel Neural Isomorphic Field, a neural abstraction of algebraic structures such as groups and fields. The elements of this neural field are embedding vectors that maintain algebraic structure during computations. Our experiments demonstrate that addition performs exceptionally well, achieving over 95 percent accuracy on key algebraic tests such as identity, closure, and associativity. In contrast, multiplication exhibits challenges, with accuracy ranging from 53 percent to 73 percent across various algebraic properties. These findings highlight the model's strengths in preserving algebraic properties under addition while identifying avenues for further improvement in handling multiplication.