Phased Instruction Fine-Tuning for Large Language Models

TL;DR

This paper introduces Phased Instruction Fine-Tuning, a progressive approach that improves large language models' adherence to instructions by sequentially training on instruction subsets ordered by difficulty, outperforming traditional one-off methods.

Contribution

The paper proposes a novel phased fine-tuning method that enhances instruction-following by gradually increasing instruction difficulty, supported by experiments on multiple models and datasets.

Findings

Phased IFT significantly outperforms One-off IFT.

Progressive training supports the hypothesis of gradual instruction alignment.

Method is simple, efficient, and broadly applicable.

Abstract

Instruction Fine-Tuning enhances pre-trained language models from basic next-word prediction to complex instruction-following. However, existing One-off Instruction Fine-Tuning (One-off IFT) method, applied on a diverse instruction, may not effectively boost models' adherence to instructions due to the simultaneous handling of varying instruction complexities. To improve this, Phased Instruction Fine-Tuning (Phased IFT) is proposed, based on the idea that learning to follow instructions is a gradual process. It assesses instruction difficulty using GPT-4, divides the instruction data into subsets of increasing difficulty, and uptrains the model sequentially on these subsets. Experiments with Llama-2 7B/13B/70B, Llama3 8/70B and Mistral-7B models using Alpaca data show that Phased IFT significantly outperforms One-off IFT, supporting the progressive alignment hypothesis and providing a simple and efficient way to enhance large language models. Codes and datasets from our experiments are freely available at https://github.com/xubuvd/PhasedSFT.