RIDE: Difficulty Evolving Perturbation with Item Response Theory for Mathematical Reasoning

TL;DR

RIDE is a novel framework that uses Item Response Theory and large language models to generate challenging, well-posed mathematical questions for more accurate evaluation of LLM reasoning abilities.

Contribution

It introduces an adversarial question-rewriting method guided by IRT and LLMs to systematically evaluate and challenge mathematical reasoning in LLMs.

Findings

Achieves an average 21.73% performance drop across 26 models.

Generates well-posed, more difficult mathematical questions.

Exposes limitations in current LLM reasoning robustness.

Abstract

Large language models (LLMs) achieve high performance on mathematical reasoning, but these results can be inflated by training data leakage or superficial pattern matching rather than genuine reasoning. To this end, an adversarial perturbation-based evaluation is needed to measure true mathematical reasoning ability. Current rule-based perturbation methods often generate ill-posed questions and impede the systematic evaluation of question difficulty and the evolution of benchmarks. To bridge this gap, we propose RIDE, a novel adversarial question-rewriting framework that leverages Item Response Theory (IRT) to rigorously measure question difficulty and to generate intrinsically more challenging, well-posed variations of mathematical problems. We employ 35 LLMs to simulate students and build a difficulty ranker from their responses. This ranker provides a reward signal during reinforcement learning and guides a question-rewriting model to reformulate existing questions across difficulty levels. Applying RIDE to competition-level mathematical benchmarks yields perturbed versions that degrade advanced LLM performance, with experiments showing an average 21.73% drop across 26 models, thereby exposing limited robustness in mathematical reasoning and confirming the validity of our evaluation approach.