FoNE: Precise Single-Token Number Embeddings via Fourier Features

TL;DR

FoNE introduces a Fourier feature-based single-token embedding for numbers in LLMs, improving efficiency and accuracy in numerical tasks by directly encoding numbers with Fourier features.

Contribution

FoNE is a novel method that maps numbers into a compact Fourier feature embedding, enabling single-token representation and superior performance on numerical tasks.

Findings

FoNE achieves 99% accuracy on 6-digit addition with 64x less data.

FoNE uses fewer tokens per number compared to subword and digit-wise embeddings.

FoNE attains 100% accuracy on large test sets for basic arithmetic operations.

Abstract

Large Language Models (LLMs) typically represent numbers using multiple tokens, which requires the model to aggregate these tokens to interpret numerical values. This fragmentation makes both training and inference less efficient and adversely affects the model's performance on number-related tasks. Inspired by the observation that pre-trained LLMs internally learn Fourier-like features for number tokens, we propose Fourier Number Embedding (FoNE), a novel method that directly maps numbers into the embedding space with their Fourier features. FoNE encodes each number as a single token with only two embedding dimensions per digit, effectively capturing numerical values without fragmentation. This compact representation accelerates both training and inference. Compared to traditional subword and digit-wise embeddings, FoNE not only reduces computational overhead but also achieves higher accuracy across various numerical tasks including addition, subtraction and multiplication. On 6-digit decimal addition, FoNE requires 64$\times$ less data to achieve 99% accuracy than subword and digit-wise embeddings while using 3$\times$ and 6$\times$ fewer tokens per number, respectively. Furthermore, FoNE is the only method that yields 100% accuracy on over 100,000 test examples for addition, subtraction, and multiplication. The codes and visualization are available at https://fouriernumber.github.io/.