Large Language Models and Mathematical Reasoning Failures

TL;DR

This study assesses the mathematical reasoning of large language models using high-school-level problems, revealing persistent reasoning failures despite high accuracy in answers, especially in complex, multi-step, or real-world scenarios.

Contribution

It introduces a comprehensive analysis of reasoning steps in LLMs, highlighting specific failure modes and emphasizing the importance of evaluating reasoning processes beyond answer correctness.

Findings

All models exhibit reasoning errors in spatial, strategic, and arithmetic tasks.

Newer models perform better but still struggle with multi-step deduction.

Models often produce correct answers through flawed logic.

Abstract

This paper investigates the mathematical reasoning capabilities of large language models (LLMs) using 50 newly constructed high-school-level word problems. Unlike prior studies that focus solely on answer correctness, we rigorously analyze both final answers and solution steps to identify reasoning failures. Evaluating eight state-of-the-art models - including Mixtral, Llama, Gemini, GPT-4o, and OpenAI's o1 variants - we find that while newer models (e.g., o3-mini, deepseek-r1) achieve higher accuracy, all models exhibit errors in spatial reasoning, strategic planning, and arithmetic, sometimes producing correct answers through flawed logic. Common failure modes include unwarranted assumptions, over-reliance on numerical patterns, and difficulty translating physical intuition into mathematical steps. Manual analysis reveals that models struggle with problems requiring multi-step deduction or real-world knowledge, despite possessing broad mathematical knowledge. Our results underscore the importance of evaluating reasoning processes, not just answers, and caution against overestimating LLMs' problem-solving proficiency. The study highlights persistent gaps in LLMs' generalization abilities, emphasizing the need for targeted improvements in structured reasoning and constraint handling.