Learning by Analogy: Enhancing Few-Shot Prompting for Math Word Problem Solving with Computational Graph-Based Retrieval

TL;DR

This paper introduces a method that retrieves similar problems based on computational graphs to improve large language models' reasoning in solving math word problems, significantly boosting performance.

Contribution

It proposes a novel retrieval-based approach using computational graph similarity to enhance few-shot prompting for math problem solving in LLMs.

Findings

Up to 6.7% average performance improvement across datasets

Effective use of computational graph retrieval for analogy-based prompting

Demonstrates potential for reasoning enhancement in LLMs

Abstract

Large language models (LLMs) are known to struggle with complicated reasoning tasks such as math word problems (MWPs). In this paper, we present how analogy from similarly structured questions can improve LLMs' problem-solving capabilities for MWPs. Specifically, we rely on the retrieval of problems with similar computational graphs to the given question to serve as exemplars in the prompt, providing the correct reasoning path for the generation model to refer to. Empirical results across six math word problem datasets demonstrate the effectiveness of our proposed method, which achieves a significant improvement of up to 6.7 percent on average in absolute value, compared to baseline methods. These results highlight our method's potential in addressing the reasoning challenges in current LLMs.