AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modeling

TL;DR

AceMath introduces advanced math reasoning models with specialized training and reward systems, significantly improving performance on complex math problems and setting new benchmarks in math reasoning AI.

Contribution

The paper presents a novel instruction-tuning process and a comprehensive reward model for math reasoning, outperforming existing models and establishing new evaluation benchmarks.

Findings

AceMath-72B-Instruct outperforms GPT-4o and Claude-3.5 Sonnet in math reasoning.

AceMath-72B-RM surpasses state-of-the-art reward models.

Combining AceMath models yields the highest rm@8 scores across benchmarks.

Abstract

In this paper, we introduce AceMath, a suite of frontier math models that excel in solving complex math problems, along with highly effective reward models capable of evaluating generated solutions and reliably identifying the correct ones. To develop the instruction-tuned math models, we propose a supervised fine-tuning (SFT) process that first achieves competitive performance across general domains, followed by targeted fine-tuning for the math domain using a carefully curated set of prompts and synthetically generated responses. The resulting model, AceMath-72B-Instruct greatly outperforms Qwen2.5-Math-72B-Instruct, GPT-4o and Claude-3.5 Sonnet. To develop math-specialized reward model, we first construct AceMath-RewardBench, a comprehensive and robust benchmark for evaluating math reward models across diverse problems and difficulty levels. After that, we present a systematic approach to build our math reward models. The resulting model, AceMath-72B-RM, consistently outperforms state-of-the-art reward models. Furthermore, when combining AceMath-72B-Instruct with AceMath-72B-RM, we achieve the highest average rm@8 score across the math reasoning benchmarks. We release model weights, training data, and evaluation benchmarks at: https://research.nvidia.com/labs/adlr/acemath