SPARQ: Synthetic Problem Generation for Reasoning via Quality-Diversity Algorithms

TL;DR

SPARQ introduces a novel method for generating high-quality, diverse synthetic math problems using a single model and quality-diversity algorithms, significantly enhancing reasoning capabilities and generalization of language models.

Contribution

The paper presents SPARQ, a new approach for scalable synthetic problem generation that improves model performance and generalization by filtering for difficulty and diversity.

Findings

Filtering by problem difficulty improves in-distribution performance.

Diverse synthetic data enhances out-of-distribution robustness.

Scaling laws exist for synthetic problems, benefiting model generalization.

Abstract

Large language model (LLM) driven synthetic data generation has emerged as a powerful method for improving model reasoning capabilities. However, most methods either distill large state-of-the-art models into small students or use natural ground-truth problem statements to guarantee problem statement quality. This limits the scalability of these approaches to more complex and diverse problem domains. To address this, we present SPARQ: Synthetic Problem Generation for Reasoning via Quality-Diversity Algorithms, a novel approach for generating high-quality and diverse synthetic math problem and solution pairs using only a single model by measuring a problem's solve-rate: a proxy for problem difficulty. Starting from a seed dataset of 7.5K samples, we generate over 20 million new problem-solution pairs. We show that filtering the generated data by difficulty and then fine-tuning the same model on the resulting data improves relative model performance by up to 24\%. Additionally, we conduct ablations studying the impact of synthetic data quantity, quality and diversity on model generalization. We find that higher quality, as measured by problem difficulty, facilitates better in-distribution performance. Further, while generating diverse synthetic data does not as strongly benefit in-distribution performance, filtering for more diverse data facilitates more robust OOD generalization. We also confirm the existence of model and data scaling laws for synthetically generated problems, which positively benefit downstream model generalization.