MDPO: Multi-Granularity Direct Preference Optimization for Mathematical Reasoning

TL;DR

This paper introduces MDPO, a multi-granularity preference optimization method that significantly improves mathematical reasoning in LLMs by addressing limitations of existing DPO techniques across different reasoning levels.

Contribution

The paper proposes a novel multi-granularity approach to optimize LLMs for mathematical reasoning, unifying training objectives across solution, inference, and step levels to enhance correctness and computational ability.

Findings

Achieved 1.7% and 0.9% improvements on GSM8K with Qwen2 and Llama3.

Achieved 2.3% and 1.2% improvements on MATH dataset.

Outperformed existing DPO variants in experiments.

Abstract

Mathematical reasoning presents a significant challenge for Large Language Models (LLMs) as it requires ensuring the correctness of each reasoning step. Researchers have been strengthening the mathematical reasoning abilities of LLMs through supervised fine-tuning, but due to the inability to suppress incorrect outputs, illusions can easily arise. Recently, Direct Preference Optimization (DPO) has been widely adopted for aligning human intent by using preference data to prevent LLMs from generating incorrect outputs. However, it has shown limited benefits in long-chain mathematical reasoning, mainly because DPO struggles to effectively capture the differences between accepted and rejected answers from preferences in long-chain data. The inconsistency between DPO training and LLMs' generation metrics also affects the effectiveness of suppressing incorrect outputs. We propose the Multi-Granularity Direct Preference Optimization (MDPO) method, optimizing the mathematical reasoning of LLMs at three granularities: Solution2Solution, Inference2Inference, and Step2Step. Solution2Solution focuses on the correctness of entire long-chain reasoning; Inference2Inference concentrates on logical reasoning between steps; Step2Step corrects computational errors in steps, enhancing the computational capabilities of LLMs. Additionally, we unify the training objectives of the three granularities to align with the generation metrics. We conducted experiments on the open-source models Qwen2 and Llama3, achieving improvements of 1.7% and 0.9% on the GSM8K dataset, and 2.3% and 1.2% on the MATH dataset, outperforming DPO and other DPO variant methods. Furthermore, we also provide a pipeline for constructing MDPO training data that is simple and does not require manual annotation costs.