DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving

TL;DR

DART-Math introduces a difficulty-aware rejection tuning method that enhances mathematical problem-solving models by focusing on challenging queries, resulting in superior performance using smaller, publicly available datasets and models.

Contribution

The paper presents DART, a novel method that prioritizes difficult queries during data synthesis, creating smaller, more effective datasets for training mathematical reasoning models without proprietary data.

Findings

DART-Math outperforms previous methods on 6 benchmarks.

Models trained with DART datasets outperform those trained with larger, less focused datasets.

Synthetic datasets created with DART are the most cost-effective publicly available resources.

Abstract

Solving mathematical problems requires advanced reasoning abilities and presents notable challenges for large language models. Previous works usually synthesize data from proprietary models to augment existing datasets, followed by instruction tuning to achieve top-tier results. However, our analysis of these datasets reveals severe biases towards easy queries, with frequent failures to generate any correct response for the most challenging queries. Hypothesizing that difficult queries are crucial to learn complex reasoning, we propose Difficulty-Aware Rejection Tuning (DART), a method that allocates difficult queries more trials during the synthesis phase, enabling more extensive training on difficult samples. Utilizing DART, we have created new datasets for mathematical problem-solving that focus more on difficult queries and are substantially smaller than previous ones. Remarkably, our synthesis process solely relies on a 7B-sized open-weight model, without reliance on the commonly used proprietary GPT-4. We fine-tune various base models on our datasets ranging from 7B to 70B in size, resulting in a series of strong models called DART-MATH. In comprehensive in-domain and out-of-domain evaluation on 6 mathematical benchmarks, DART-MATH outperforms vanilla rejection tuning significantly, being superior or comparable to previous arts, despite using much smaller datasets and no proprietary models. Furthermore, our results position our synthetic datasets as the most effective and cost-efficient publicly available resources for advancing mathematical problem-solving.