Self-Training with Direct Preference Optimization Improves Chain-of-Thought Reasoning

TL;DR

This paper introduces a self-training method enhanced with Direct Preference Optimization to improve the reasoning capabilities of small language models, making them more accurate and scalable for mathematical reasoning tasks.

Contribution

The work demonstrates that combining self-training with DPO significantly enhances small LMs' reasoning abilities, offering a cost-effective alternative to large proprietary models.

Findings

Improved reasoning accuracy across multiple tasks

Enhanced diversity and correctness of generated reasoning

More scalable and cost-effective training approach

Abstract

Effective training of language models (LMs) for mathematical reasoning tasks demands high-quality supervised fine-tuning data. Besides obtaining annotations from human experts, a common alternative is sampling from larger and more powerful LMs. However, this knowledge distillation approach can be costly and unstable, particularly when relying on closed-source, proprietary LMs like GPT-4, whose behaviors are often unpredictable. In this work, we demonstrate that the reasoning abilities of small-scale LMs can be enhanced through self-training, a process where models learn from their own outputs. We also show that the conventional self-training can be further augmented by a preference learning algorithm called Direct Preference Optimization (DPO). By integrating DPO into self-training, we leverage preference data to guide LMs towards more accurate and diverse chain-of-thought reasoning. We evaluate our method across various mathematical reasoning tasks using different base models. Our experiments show that this approach not only improves LMs' reasoning performance but also offers a more cost-effective and scalable solution compared to relying on large proprietary LMs.