Self-Refine Instruction-Tuning for Aligning Reasoning in Language Models

TL;DR

This paper introduces a Self-refine Instruction-tuning method that enhances the reasoning abilities of smaller language models by enabling them to self-improve through preference optimization, leading to better alignment with larger models.

Contribution

The paper proposes a novel two-stage Self-refine Instruction-tuning approach that improves reasoning ability transfer and self-refinement in smaller language models.

Findings

Significant performance improvements on commonsense reasoning tasks.

Enhanced generalization in out-of-domain scenarios.

Outperforms traditional instruction-tuning methods.

Abstract

The alignments of reasoning abilities between smaller and larger Language Models are largely conducted via Supervised Fine-Tuning (SFT) using demonstrations generated from robust Large Language Models (LLMs). Although these approaches deliver more performant models, they do not show sufficiently strong generalization ability as the training only relies on the provided demonstrations. In this paper, we propose the Self-refine Instruction-tuning method that elicits Smaller Language Models to self-refine their abilities. Our approach is based on a two-stage process, where reasoning abilities are first transferred between LLMs and Small Language Models (SLMs) via Instruction-tuning on demonstrations provided by LLMs, and then the instructed models Self-refine their abilities through preference optimization strategies. In particular, the second phase operates refinement heuristics based on the Direct Preference Optimization algorithm, where the SLMs are elicited to deliver a series of reasoning paths by automatically sampling the generated responses and providing rewards using ground truths from the LLMs. Results obtained on commonsense and math reasoning tasks show that this approach significantly outperforms Instruction-tuning in both in-domain and out-domain scenarios, aligning the reasoning abilities of Smaller and Larger Language Models.