Instruction Tuning with GPT-4

TL;DR

This paper explores using GPT-4 to generate instruction-following data for fine-tuning large language models, demonstrating improved zero-shot performance and providing a new approach to data creation without human input.

Contribution

It introduces GPT-4 generated instruction data for LLM fine-tuning, showing superior performance over previous data sources and providing resources for further research.

Findings

GPT-4 generated data improves zero-shot task performance

The dataset includes 52K instructions in English and Chinese

Public release of data and code for community use

Abstract

Prior work has shown that finetuning large language models (LLMs) using machine-generated instruction-following data enables such models to achieve remarkable zero-shot capabilities on new tasks, and no human-written instructions are needed. In this paper, we present the first attempt to use GPT-4 to generate instruction-following data for LLM finetuning. Our early experiments on instruction-tuned LLaMA models show that the 52K English and Chinese instruction-following data generated by GPT-4 leads to superior zero-shot performance on new tasks to the instruction-following data generated by previous state-of-the-art models. We also collect feedback and comparison data from GPT-4 to enable a comprehensive evaluation and reward model training. We make our data generated using GPT-4 as well as our codebase publicly available.