Reverse That Number! Decoding Order Matters in Arithmetic Learning

TL;DR

This paper introduces a novel approach to arithmetic learning in LLMs by prioritizing the least significant digit, leading to improved accuracy and reduced training tokens compared to traditional step-by-step methods.

Contribution

It proposes a new digit order strategy that redefines arithmetic learning in LLMs, significantly enhancing performance and efficiency over existing methods.

Findings

Improved accuracy over SOTA methods

Reduced training tokens by two-thirds

Effective digit order prioritization enhances learning

Abstract

Recent advancements in pretraining have demonstrated that modern Large Language Models (LLMs) possess the capability to effectively learn arithmetic operations. However, despite acknowledging the significance of digit order in arithmetic computation, current methodologies predominantly rely on sequential, step-by-step approaches for teaching LLMs arithmetic, resulting in a conclusion where obtaining better performance involves fine-grained step-by-step. Diverging from this conventional path, our work introduces a novel strategy that not only reevaluates the digit order by prioritizing output from the least significant digit but also incorporates a step-by-step methodology to substantially reduce complexity. We have developed and applied this method in a comprehensive set of experiments. Compared to the previous state-of-the-art (SOTA) method, our findings reveal an overall improvement of in accuracy while requiring only a third of the tokens typically used during training. For the purpose of facilitating replication and further research, we have made our code and dataset publicly available at \url{https://anonymous.4open.science/r/RAIT-9FB7/}.