OccamLLM: Fast and Exact Language Model Arithmetic in a Single Step

TL;DR

OccamLLM introduces a novel framework that enables exact arithmetic in a single step using a symbolic approach, significantly improving speed, security, and accuracy in LLM-based arithmetic operations and benchmarks.

Contribution

The paper presents a new method combining LLM hidden states with symbolic architecture to perform exact arithmetic instantly, surpassing existing models in accuracy and efficiency.

Findings

Achieves 100% accuracy on basic arithmetic operations

Outperforms GPT-4 on mathematical benchmarks

Provides faster and more secure arithmetic computations

Abstract

Despite significant advancements in text generation and reasoning, Large Language Models (LLMs) still face challenges in accurately performing complex arithmetic operations. Language model systems often enable LLMs to generate code for arithmetic operations to achieve accurate calculations. However, this approach compromises speed and security, and fine-tuning risks the language model losing prior capabilities. We propose a framework that enables exact arithmetic in a single autoregressive step, providing faster, more secure, and more interpretable LLM systems with arithmetic capabilities. We use the hidden states of a LLM to control a symbolic architecture that performs arithmetic. Our implementation using Llama 3 with OccamNet as a symbolic model (OccamLlama) achieves 100\% accuracy on single arithmetic operations ($+,-,\times,\div,\sin{},\cos{},\log{},\exp{},\sqrt{}$), outperforming GPT 4o with and without a code interpreter. Furthermore, OccamLlama outperforms GPT 4o with and without a code interpreter on average across a range of mathematical problem solving benchmarks, demonstrating that OccamLLMs can excel in arithmetic tasks, even surpassing much larger models. We will make our code public shortly.