MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning

TL;DR

MAmmoTH introduces a series of open-source large language models tailored for general math problem-solving, leveraging a hybrid of chain-of-thought and program-of-thought rationales to significantly outperform existing models.

Contribution

The paper presents MAmmoTH, a new open-source LLM series trained on a curated math instruction dataset with hybrid rationales, achieving state-of-the-art performance on multiple math reasoning benchmarks.

Findings

MAmmoTH models outperform existing open-source models by 16-32% on nine datasets.

MAmmoTH-7B reaches 33% on MATH, surpassing WizardMath by 23%.

MAmmoTH-34B achieves 44% on MATH, exceeding GPT-4's CoT results.

Abstract

We introduce MAmmoTH, a series of open-source large language models (LLMs) specifically tailored for general math problem-solving. The MAmmoTH models are trained on MathInstruct, our meticulously curated instruction tuning dataset. MathInstruct is compiled from 13 math datasets with intermediate rationales, six of which have rationales newly curated by us. It presents a unique hybrid of chain-of-thought (CoT) and program-of-thought (PoT) rationales, and also ensures extensive coverage of diverse fields in math. The hybrid of CoT and PoT not only unleashes the potential of tool use but also allows different thought processes for different math problems. As a result, the MAmmoTH series substantially outperform existing open-source models on nine mathematical reasoning datasets across all scales with an average accuracy gain between 16% and 32%. Remarkably, our MAmmoTH-7B model reaches 33% on MATH (a competition-level dataset), which exceeds the best open-source 7B model (WizardMath) by 23%, and the MAmmoTH-34B model achieves 44% accuracy on MATH, even surpassing GPT-4's CoT result. Our work underscores the importance of diverse problem coverage and the use of hybrid rationales in developing superior math generalist models.