MIND: Math Informed syNthetic Dialogues for Pretraining LLMs

TL;DR

This paper introduces MIND, a novel method for generating math-informed synthetic dialogues to enhance the mathematical reasoning capabilities of large language models, leading to significant performance improvements across multiple benchmarks.

Contribution

The paper presents a large-scale, diverse synthetic dialogue generation approach that incorporates knowledge gaps and optimized data integration for improved LLM mathematical reasoning.

Findings

Pretraining on MIND-OWM significantly improves mathematical reasoning scores.

Incorporating knowledge gaps enhances the quality of synthetic math data.

Restructuring raw data during pretraining maximizes reasoning gains.

Abstract

The utility of synthetic data to enhance pretraining data quality and hence to improve downstream task accuracy has been widely explored in recent large language models (LLMs). Yet, these approaches fall inadequate in complex, multi-hop and mathematical reasoning tasks as the synthetic data typically fails to add complementary knowledge to the existing raw corpus. In this work, we propose a novel large-scale and diverse Math Informed syNthetic Dialogue (MIND) generation method that improves the mathematical reasoning ability of LLMs. Specifically, using MIND, we generate synthetic conversations based on OpenWebMath (OWM), resulting in a new math corpus, MIND-OWM. Our experiments with different conversational settings reveal that incorporating knowledge gaps between dialog participants is essential for generating high-quality math data. We further identify an effective way to format and integrate synthetic and raw data during pretraining to maximize the gain in mathematical reasoning, emphasizing the need to restructure raw data rather than use it as-is. Compared to pretraining just on raw data, a model pretrained on MIND-OWM shows significant boost in mathematical reasoning (GSM8K: +13.42%, MATH: +2.30%), including superior performance in specialized knowledge (MMLU: +4.55%, MMLU-STEM: +4.28%) and general purpose reasoning tasks (GENERAL REASONING: +2.51%).