Evaluation of LLMs for mathematical problem solving

TL;DR

This study evaluates three large language models on mathematical problem solving across various datasets, analyzing their strengths and weaknesses in reasoning, explanation, and accuracy.

Contribution

It introduces a comprehensive five-dimensional assessment framework for LLMs in math problem solving and provides detailed performance comparisons of GPT-4o, DeepSeek-V3, and Gemini-2.0.

Findings

GPT-4o is the most stable across datasets, especially in high-level questions.

DeepSeek-V3 excels in structured domains like optimization.

Gemini-2.0 has strong linguistic understanding but struggles with multi-step reasoning.

Abstract

Large Language Models (LLMs) have shown impressive performance on a range of educational tasks, but are still understudied for their potential to solve mathematical problems. In this study, we compare three prominent LLMs, including GPT-4o, DeepSeek-V3, and Gemini-2.0, on three mathematics datasets of varying complexities (GSM8K, MATH500, and MIT Open Courseware datasets). We take a five-dimensional approach based on the Structured Chain-of-Thought (SCoT) framework to assess final answer correctness, step completeness, step validity, intermediate calculation accuracy, and problem comprehension. The results show that GPT-4o is the most stable and consistent in performance across all the datasets, but particularly it performs outstandingly in high-level questions of the MIT Open Courseware dataset. DeepSeek-V3 is competitively strong in well-structured domains such as optimisation, but suffers from fluctuations in accuracy in statistical inference tasks. Gemini-2.0 shows strong linguistic understanding and clarity in well-structured problems but performs poorly in multi-step reasoning and symbolic logic. Our error analysis reveals particular deficits in each model: GPT-4o is at times lacking in sufficient explanation or precision; DeepSeek-V3 leaves out intermediate steps; and Gemini-2.0 is less flexible in mathematical reasoning in higher dimensions.