Can LLMs Solve longer Math Word Problems Better?

TL;DR

This paper investigates how well large language models can solve longer math word problems, introduces new datasets and metrics, and proposes methods to improve their reasoning capabilities with extended narratives.

Contribution

It introduces the E-GSM dataset, new evaluation metrics for context length generalizability, and tailored prompting and fine-tuning techniques to enhance LLM performance on lengthy MWPs.

Findings

Existing LLMs show limited ability to handle longer MWPs.

Proposed methods improve LLM performance on extended narratives.

Enhanced methods also generalize across other MWP benchmarks.

Abstract

Math Word Problems (MWPs) play a vital role in assessing the capabilities of Large Language Models (LLMs), yet current research primarily focuses on questions with concise contexts. The impact of longer contexts on mathematical reasoning remains under-explored. This study pioneers the investigation of Context Length Generalizability (CoLeG), which refers to the ability of LLMs to solve MWPs with extended narratives. We introduce Extended Grade-School Math (E-GSM), a collection of MWPs featuring lengthy narratives, and propose two novel metrics to evaluate the efficacy and resilience of LLMs in tackling these problems. Our analysis of existing zero-shot prompting techniques with proprietary LLMs along with open-source LLMs reveals a general deficiency in CoLeG. To alleviate these issues, we propose tailored approaches for different categories of LLMs. For proprietary LLMs, we introduce a new instructional prompt designed to mitigate the impact of long contexts. For open-source LLMs, we develop a novel auxiliary task for fine-tuning to enhance CoLeG. Our comprehensive results demonstrate the effectiveness of our proposed methods, showing improved performance on E-GSM. Additionally, we conduct an in-depth analysis to differentiate the effects of semantic understanding and reasoning efficacy, showing that our methods improves the latter. We also establish the generalizability of our methods across several other MWP benchmarks. Our findings highlight the limitations of current LLMs and offer practical solutions correspondingly, paving the way for further exploration of model generalizability and training methodologies.