Large Language Models Don't Make Sense of Word Problems. A Scoping Review from a Mathematics Education Perspective

TL;DR

This paper reviews the capabilities of large language models in solving mathematical word problems, finding they excel at superficial problem-solving but struggle with real-world context understanding, limiting their educational usefulness.

Contribution

It provides a comprehensive scoping review, including technical, literature, and empirical analyses, revealing LLMs' superficial understanding of word problems from a mathematics education perspective.

Findings

LLMs solve s-problems with near-perfect accuracy.

Most word problems in research lack real-world context.

LLMs struggle with problems involving real-world or nonsensical contexts.

Abstract

The progress of Large Language Models (LLMs) like ChatGPT raises the question of how they can be integrated into education. One hope is that they can support mathematics learning, including word-problem solving. Since LLMs can handle textual input with ease, they appear well-suited for solving mathematical word problems. Yet their real competence, whether they can make sense of the real-world context, and the implications for classrooms remain unclear. We conducted a scoping review from a mathematics-education perspective, including three parts: a technical overview, a systematic review of word problems used in research, and a state-of-the-art empirical evaluation of LLMs on mathematical word problems. First, in the technical overview, we contrast the conceptualization of word problems and their solution processes between LLMs and students. In computer-science research this is typically labeled mathematical reasoning, a term that does not align with usage in mathematics education. Second, our literature review of 213 studies shows that the most popular word-problem corpora are dominated by s-problems, which do not require a consideration of realities of their real-world context. Finally, our evaluation of GPT-3.5-turbo, GPT-4o-mini, GPT-4.1, o3, and GPT-5 on 287 word problems shows that most recent LLMs solve these s-problems with near-perfect accuracy, including a perfect score on 20 problems from PISA. LLMs still showed weaknesses in tackling problems where the real-world context is problematic or non-sensical. In sum, we argue based on all three aspects that LLMs have mastered a superficial solution process but do not make sense of word problems, which potentially limits their value as instructional tools in mathematics classrooms.