U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills in LLMs

TL;DR

U-MATH introduces a comprehensive university-level benchmark with diverse, open-ended problems across multiple subjects, highlighting current LLMs' limitations especially in multimodal reasoning and solution judgment.

Contribution

The paper presents U-MATH, a new large-scale, diverse benchmark for evaluating LLMs on university-level mathematics, including multimodal problems and a solution judgment dataset.

Findings

LLMs perform well on textual tasks with up to 93.1% accuracy.

Multimodal reasoning remains challenging, with only 58.5% accuracy.

Solution judgment accuracy peaks at 90.1% F1-score, still imperfect.

Abstract

The current evaluation of mathematical skills in LLMs is limited, as existing benchmarks are either relatively small, primarily focus on elementary and high-school problems, or lack diversity in topics. Additionally, the inclusion of visual elements in tasks remains largely under-explored. To address these gaps, we introduce U-MATH, a novel benchmark of 1,100 unpublished open-ended university-level problems sourced from teaching materials. It is balanced across six core subjects, with 20% of multimodal problems. Given the open-ended nature of U-MATH problems, we employ an LLM to judge the correctness of generated solutions. To this end, we release $\mu$-MATH, a dataset to evaluate the LLMs' capabilities in judging solutions. Benchmarking leading LLMs reveals marked limitations in multi-modal reasoning, with maximum accuracy reaching 93.1\% on textual tasks but only 58.5\% on visual ones. Furthermore, solution judgment proves challenging, requiring the most advanced models to achieve meaningfully high performance, even still peaking at an imperfect F1-score of 90.1\%.