REInstruct: Building Instruction Data from Unlabeled Corpus

TL;DR

REInstruct presents a scalable method to automatically generate high-quality instruction data from unlabeled corpora, reducing reliance on proprietary models and human annotation, and improving instruction tuning for large language models.

Contribution

It introduces a novel approach to create instruction datasets from unlabeled texts without heavy dependence on proprietary LLMs or manual labeling.

Findings

Achieves 65.41% win rate on AlpacaEval leaderboard.

Outperforms other open-source instruction data methods.

Effectively enhances Llama-7b performance.

Abstract

Manually annotating instruction data for large language models is difficult, costly, and hard to scale. Meanwhile, current automatic annotation methods typically rely on distilling synthetic data from proprietary LLMs, which not only limits the upper bound of the quality of the instruction data but also raises potential copyright issues. In this paper, we propose REInstruct, a simple and scalable method to automatically build instruction data from an unlabeled corpus without heavy reliance on proprietary LLMs and human annotation. Specifically, REInstruct first selects a subset of unlabeled texts that potentially contain well-structured helpful and insightful content and then generates instructions for these texts. To generate accurate and relevant responses for effective and robust training, REInstruct further proposes a rewriting-based approach to improve the quality of the generated instruction data. By training Llama-7b on a combination of 3k seed data and 32k synthetic data from REInstruct, fine-tuned model achieves a 65.41\% win rate on AlpacaEval leaderboard against text-davinci-003, outperforming other open-source, non-distilled instruction data construction methods. The code is publicly available at \url{https://github.com/cs32963/REInstruct}.