Mathador-LM: A Dynamic Benchmark for Mathematical Reasoning on Large Language Models

TL;DR

Mathador-LM is a new dynamic benchmark for assessing mathematical reasoning in large language models, designed to prevent test data leakage and reveal models' struggles with complex math problems compared to human performance.

Contribution

We introduce Mathador-LM, a novel dynamic benchmark that evaluates LLMs' mathematical reasoning while addressing test leakage issues and providing comprehensive model assessments.

Findings

Contemporary LLMs perform poorly on Mathador-LM, scoring below average 3rd graders.

Mathador-LM's dynamic instance generation prevents test set leakage.

Models show significant gaps in mathematical reasoning compared to human performance.

Abstract

We introduce Mathador-LM, a new benchmark for evaluating the mathematical reasoning on large language models (LLMs), combining ruleset interpretation, planning, and problem-solving. This benchmark is inspired by the Mathador game, where the objective is to reach a target number using basic arithmetic operations on a given set of base numbers, following a simple set of rules. We show that, across leading LLMs, we obtain stable average performance while generating benchmark instances \emph{dynamically}, following a target difficulty level. Thus, our benchmark alleviates concerns about test-set leakage into training data, an issue that often undermines popular benchmarks. Additionally, we conduct a comprehensive evaluation of both open and closed-source state-of-the-art LLMs on Mathador-LM. Our findings reveal that contemporary models struggle with Mathador-LM, scoring significantly lower than average 3rd graders. This stands in stark contrast to their strong performance on popular mathematical reasoning benchmarks. The implementation of Mathador-LM benchmark is available at \href{https://github.com/IST-DASLab/Mathador-LM}{github.com/IST-DASLab/Mathador-LM}.