IGC: Integrating a Gated Calculator into an LLM to Solve Arithmetic Tasks Reliably and Efficiently

TL;DR

The paper introduces the Integrated Gated Calculator (IGC), a novel module that enables LLMs to perform arithmetic reliably and efficiently by emulating a calculator on the GPU, surpassing state-of-the-art models on the BigBench Arithmetic benchmark.

Contribution

The paper presents IGC, a new module that allows LLMs to perform arithmetic internally without external tools, improving accuracy and efficiency over previous methods.

Findings

Achieves 98-99% accuracy on arithmetic tasks

Outperforms all models on BigBench Arithmetic benchmark

Successfully handles complex tasks like multiplication

Abstract

Solving arithmetic tasks is a simple and fundamental skill, yet modern Large Language Models (LLMs) have great difficulty with them. We introduce the Integrated Gated Calculator (IGC), a module that enables LLMs to perform arithmetic by emulating a calculator on the GPU. We finetune a Llama model with our module and test it on the BigBench Arithmetic benchmark, where it beats the State of the Art, outperforming all models on the benchmark, including models almost two orders of magnitude larger. Our approach takes only a single iteration to run and requires no external tools. It performs arithmetic operations entirely inside the LLM without the need to produce intermediate tokens. It is computationally efficient, interpretable, and avoids side-effects on tasks that do not require arithmetic operations. It reliably achieves 98\% to 99\% accuracy across multiple training runs and for all subtasks, including the substantially harder subtask of multiplication, which was previously unsolved.