Executing Arithmetic: Fine-Tuning Large Language Models as Turing Machines

TL;DR

This paper introduces CAEF, a framework that enables large language models to perform arithmetic by emulating Turing Machines, significantly improving their ability to generalize and accurately compute complex operations.

Contribution

The paper presents a scalable framework allowing LLMs to learn step-by-step arithmetic execution as Turing Machines, enhancing their computational understanding and generalization capabilities.

Findings

Achieves nearly 100% accuracy on seven arithmetic operations

Supports computations with operands up to 100 digits

Outperforms GPT-4o in complex arithmetic tasks

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide range of natural language processing and reasoning tasks. However, their performance in the foundational domain of arithmetic remains unsatisfactory. When dealing with arithmetic tasks, LLMs often memorize specific examples rather than learning the underlying computational logic, limiting their ability to generalize to new problems. In this paper, we propose a Composable Arithmetic Execution Framework (CAEF) that enables LLMs to learn to execute step-by-step computations by emulating Turing Machines, thereby gaining a genuine understanding of computational logic. Moreover, the proposed framework is highly scalable, allowing composing learned operators to significantly reduce the difficulty of learning complex operators. In our evaluation, CAEF achieves nearly 100% accuracy across seven common mathematical operations on the LLaMA 3.1-8B model, effectively supporting computations involving operands with up to 100 digits, a level where GPT-4o falls short noticeably in some settings.