LLMs for Mathematical Modeling: Towards Bridging the Gap between Natural and Mathematical Languages

TL;DR

This paper introduces Mamo, a benchmark for evaluating LLMs' ability to construct mathematical models, revealing current limitations and performance differences across model sizes and types in complex mathematical reasoning tasks.

Contribution

The paper presents a novel process-oriented evaluation framework and a comprehensive benchmark, Mamo, for assessing LLMs' mathematical modeling capabilities.

Findings

Larger models perform better on complex tasks

Open-source models are competitive on simpler problems

All models struggle with advanced mathematical modeling

Abstract

Large Language Models (LLMs) have demonstrated strong performance across various natural language processing tasks, yet their proficiency in mathematical reasoning remains a key challenge. Addressing the gap between natural and mathematical language requires advanced reasoning capabilities, approaching those of Artificial General Intelligence (AGI). However, the evaluation remains challenging, as perfectly representing reality is inherently elusive, and traditional methods like manual or direct comparison of mathematical statements (Ramamonjison et al., 2023) are insufficient for assessing true modeling ability. We propose a process-oriented framework to evaluate LLMs' ability to construct mathematical models, using solvers to compare outputs with ground truth. Introducing Mamo, a benchmark with 1,209 questions covering ordinary differential equations, linear programming, and mixed-integer linear programming, we enable automatic evaluation of modeling accuracy. The results show that existing LLMs struggle with complex mathematical modeling tasks, with larger models demonstrating superior performance, while open-source models remain competitive in simpler cases but still fall short of proprietary models in more challenging problems.