FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale

TL;DR

This paper introduces FineInstructions, a large synthetic instruction dataset generated from pre-training data, enabling effective instruction-based pre-training of language models that improves response quality on downstream tasks.

Contribution

The authors propose a scalable method to generate billions of synthetic instruction-response pairs from pre-training data, enhancing instruction-based pre-training of language models.

Findings

Pre-training on FineInstructions outperforms standard pre-training on benchmarks.

Synthetic data improves model's ability to respond to user prompts.

Method enables large-scale instruction tuning without extensive human-labeled data.

Abstract

Due to limited supervised training data, large language models (LLMs) are typically pre-trained via a self-supervised "predict the next word" objective on a vast amount of unstructured text data. To make the resulting model useful to users, it is further trained on a far smaller amount of "instruction-tuning" data comprised of supervised training examples of instructions and responses. To overcome the limited amount of supervised data, we propose a procedure that can transform the knowledge in internet-scale pre-training documents into billions of synthetic instruction and answer training pairs. The resulting dataset, called FineInstructions, uses ~18M instruction templates created from real user-written queries and prompts. These instruction templates are matched to and instantiated with human-written source documents from unstructured pre-training corpora. With "supervised" synthetic training data generated at this scale, an LLM can be pre-trained from scratch solely with the instruction-tuning objective, which is far more in-distribution with the expected downstream usage of LLMs (responding to user prompts). We conduct controlled token-for-token training experiments and find pre-training on FineInstructions outperforms standard pre-training and other proposed synthetic pre-training techniques on standard benchmarks measuring free-form response quality. Our resources can be found at https://huggingface.co/fineinstructions .