System-2 Mathematical Reasoning via Enriched Instruction Tuning

TL;DR

This paper introduces Enriched Instruction Tuning (EIT), a novel method that enhances large language models' mathematical reasoning by creating detailed reasoning trajectories through human and AI feedback, significantly improving their problem-solving accuracy.

Contribution

The paper presents EIT, a new fine-tuning approach that enriches reasoning data with detailed plans and steps, outperforming existing methods in mathematical problem-solving accuracy.

Findings

EIT achieves 84.1% accuracy on GSM8K.

EIT reaches 32.5% accuracy on MATH.

EIT surpasses state-of-the-art fine-tuning and prompting methods.

Abstract

Solving complex mathematical problems via system-2 reasoning is a natural human skill, yet it remains a significant challenge for current large language models (LLMs). We identify the scarcity of deliberate multi-step reasoning data as a primary limiting factor. To this end, we introduce Enriched Instruction Tuning (EIT), a method that enriches existing human-annotated mathematical datasets by synergizing human and AI feedback to create fine-grained reasoning trajectories. These datasets are then used to fine-tune open-source LLMs, enhancing their mathematical reasoning abilities without reliance on any symbolic verification program. Concretely, EIT is composed of two critical steps: Enriching with Reasoning Plan (ERP) and Enriching with Reasoning Step (ERS). The former generates a high-level plan that breaks down complex instructions into a sequence of simpler objectives, while ERS fills in reasoning contexts often overlooked by human annotators, creating a smoother reasoning trajectory for LLM fine-tuning. Unlike existing CoT prompting methods that generate reasoning chains only depending on LLM's internal knowledge, our method leverages human-annotated initial answers as ``meta-knowledge'' to help LLMs generate more detailed and precise reasoning processes, leading to a more trustworthy LLM expert for complex mathematical problems. In experiments, EIT achieves an accuracy of 84.1% on GSM8K and 32.5% on MATH, surpassing state-of-the-art fine-tuning and prompting methods, and even matching the performance of tool-augmented methods.