MARGE: Improving Math Reasoning for LLMs with Guided Exploration

TL;DR

MARGE is a novel method that enhances mathematical reasoning in large language models by guided exploration of intermediate reasoning states, leading to improved accuracy and diversity without extra annotations.

Contribution

MARGE introduces hit-guided exploration to systematically improve reasoning exploration and credit assignment in LLMs, outperforming prior methods without external annotations.

Findings

Significant improvement in reasoning accuracy across benchmarks.

Enhanced exploration diversity without trade-offs.

Effective in multiple backbone models.

Abstract

Large Language Models (LLMs) exhibit strong potential in mathematical reasoning, yet their effectiveness is often limited by a shortage of high-quality queries. This limitation necessitates scaling up computational responses through self-generated data, yet current methods struggle due to spurious correlated data caused by ineffective exploration across all reasoning stages. To address such challenge, we introduce \textbf{MARGE}: Improving \textbf{Ma}th \textbf{R}easoning with \textbf{G}uided \textbf{E}xploration, a novel method to address this issue and enhance mathematical reasoning through hit-guided exploration. MARGE systematically explores intermediate reasoning states derived from self-generated solutions, enabling adequate exploration and improved credit assignment throughout the reasoning process. Through extensive experiments across multiple backbone models and benchmarks, we demonstrate that MARGE significantly improves reasoning capabilities without requiring external annotations or training additional value models. Notably, MARGE improves both single-shot accuracy and exploration diversity, mitigating a common trade-off in alignment methods. These results demonstrate MARGE's effectiveness in enhancing mathematical reasoning capabilities and unlocking the potential of scaling self-generated training data. Our code and models are available at \href{https://github.com/georgao35/MARGE}{this link}.