Non-instructional Fine-tuning: Enabling Instruction-Following Capabilities in Pre-trained Language Models without Instruction-Following Data

TL;DR

This paper demonstrates that pre-trained language models can acquire instruction-following abilities through fine-tuning on non-instructional data generated by GPT models, challenging the necessity of explicit instruction data.

Contribution

It introduces a novel method of using non-instructional text as training data to enable instruction-following in LLMs, bypassing the need for instruction-specific datasets.

Findings

Models fine-tuned on non-instructional data gain instruction-following capabilities.

Non-instructional data improves performance of models already fine-tuned or aligned.

LLaMA-3-70B-Instruct matches LLaMA-3.1-70B-Instruct on benchmark.

Abstract

Instruction fine-tuning is crucial for today's large language models (LLMs) to learn to follow instructions and align with human preferences. Conventionally, supervised data, including the instruction and the correct response, is required for instruction fine-tuning. To obtain such data, some researchers prompted well-trained models like GPT-4 to generate instructions and correct responses. In this paper, we propose a novel approach that uses the first half of a random text from OpenWebText as the instruction and GPT-3.5-turbo or GPT-4-turbo to complete the text as the response. Despite the data being "non-instructional", we found that pre-trained LLMs fine-tuned on this data can gain instruction-following capabilities. This observation is verified by fine-tuning several well-known pre-trained LLMs (e.g., LLaMA-2-7B, LLaMA-3-8B, LLaMA-3-70B, Mistral-7B-v0.1). The "non-instructional data" also improved some models that underwent supervised fine-tuning and human preference alignment. Our LLaMA-3-70B-Instruct fine-tuned through "non-instructional data" is comparable with LLaMA-3.1-70B-Instruct on the Arena Hard leaderboard. We analyzed the "non-instructional data" and ensured it is devoid of content related to instruction fine-tuning. Our findings will inspire further investigation into how to develop instruction-following capabilities without explicit instruction-related data.