Exploring Equation as a Better Intermediate Meaning Representation for Numerical Reasoning

TL;DR

This paper proposes using equations as intermediate meaning representations for numerical reasoning, providing theoretical proof of their advantages and a method to improve their generation accuracy with large language models, leading to better performance on benchmark datasets.

Contribution

It introduces a theoretical comparison of equations versus programs as IMRs and presents a novel method, Bridge, to enhance equation generation accuracy in LLMs for numerical reasoning.

Findings

Achieved 2.2% accuracy improvement on GSM8K dataset

Demonstrated theoretical superiority of equations as IMRs

Enhanced equation generation accuracy with the Bridge method

Abstract

Numerical reasoning is vital for natural language processing models to understand and process numerical information in real-world scenarios. Most current methods first generate the Intermediate Meaning Representations (IMRs) of questions and then generate answers. Current SOTA methods generate programs as IMRs with large language models (LLMs). Intuitively, equations have fewer restrictions and closer semantics to the question than programs, leading to higher generation accuracy. However, current LLMs generate equations worse than programs, where we assume that the equation data is rare in pre-training data compared to programs. So in this paper, we try to use equations as IMRs to solve the numerical reasoning task by addressing two problems: (1) Theoretically, how to prove that the equation is an IMR with higher generation accuracy than programs; (2) Empirically, how to improve the generation accuracy of equations with LLMs. For the first problem, we propose and prove a proposition to theoretically compare the generation accuracy of different IMRs. For the second problem, we present a method called Boosting Numerical Reason\textbfing by Decomposing the Generation of Equations (Bridge), which can improve the accuracy of LLMs in generating equations as IMRs by reducing the tendency of generating constant expressions and programs. Our method improves the performance by 2.2%, 0.9%, and 1.7% on GSM8K, SVAMP, and Algebra datasets compared to the previous state-of-the-art methods under the single reasoning path setting. Our codes and prompts are released in https://github.com/zirui-HIT/Bridge_for_Numerical_Reasoning.