RoMath: A Mathematical Reasoning Benchmark in Romanian

TL;DR

RoMath is a new Romanian mathematical reasoning benchmark designed to evaluate and improve AI models' understanding of informal mathematical language in a low-resource language, addressing the gap in multilingual mathematical AI resources.

Contribution

The paper introduces RoMath, the first Romanian mathematical reasoning benchmark with diverse subsets, and benchmarks open-weight models to highlight the need for resources in underrepresented languages.

Findings

Open-weight models show varied performance on RoMath

RoMath covers multiple difficulty levels and domains

Resources for low-resource languages are essential for multilingual AI

Abstract

Mathematics has long been conveyed through natural language, primarily for human understanding. With the rise of mechanized mathematics and proof assistants, there is a growing need to understand informal mathematical text, yet most existing benchmarks focus solely on English, overlooking other languages. This paper introduces RoMath, a Romanian mathematical reasoning benchmark suite comprising three subsets: Baccalaureate, Competitions and Synthetic, which cover a range of mathematical domains and difficulty levels, aiming to improve non-English language models and promote multilingual AI development. By focusing on Romanian, a low-resource language with unique linguistic features, RoMath addresses the limitations of Anglo-centric models and emphasizes the need for dedicated resources beyond simple automatic translation. We benchmark several open-weight language models, highlighting the importance of creating resources for underrepresented languages. Code and datasets are be made available.