Automatically Generating Hard Math Problems from Hypothesis-Driven Error Analysis

TL;DR

This paper introduces an AI-driven pipeline that automatically generates math problems targeting LLM weaknesses by analyzing hypotheses, significantly reducing LLM accuracy on generated problems and adaptable to other domains.

Contribution

It presents a novel method for automatic, hypothesis-driven math benchmark generation that identifies specific skills LLMs struggle with, improving scalability and domain adaptability.

Findings

Generated problems from high-accuracy hypotheses are more challenging for LLMs.

Llama-3.3-70B-Instruct's accuracy drops to 45% on generated problems.

Pipeline can be applied beyond math to other LLM capabilities.

Abstract

Numerous math benchmarks exist to evaluate LLMs' mathematical capabilities. However, most involve extensive manual effort and are difficult to scale. Consequently, they cannot keep pace with LLM development or easily provide new instances to mitigate overfitting. Some researchers have proposed automatic benchmark generation methods, but few focus on identifying the specific math concepts and skills on which LLMs are error-prone, and most can only generate category-specific benchmarks. To address these limitations, we propose a new math benchmark generation pipeline that uses AI-generated hypotheses to identify the specific math concepts and skills that LLMs struggle with, and then generates new benchmark problems targeting these weaknesses. Experiments show that hypothesis accuracy positively correlates with the difficulty of the generated problems: problems generated from the most accurate hypotheses reduce Llama-3.3-70B-Instruct's accuracy to as low as 45%, compared to 77% on the original MATH benchmark. Furthermore, our pipeline is highly adaptable and can be applied beyond math to explore a wide range of LLM capabilities, making it a valuable tool for investigating how LLMs perform across different domains.