Scaling Towards the Information Boundary of Instruction Sets: The Infinity Instruct Subject Technical Report

TL;DR

This paper introduces a systematic framework for constructing high-quality instruction datasets that expand coverage and depth, significantly improving large models' instruction-following abilities and generalization.

Contribution

It presents a novel iterative data construction framework integrating hierarchical tagging, seed selection, synthesis, and diagnosis, leading to the creation of the Infinity Instruct Subject dataset.

Findings

The dataset contains approximately 1.5 million instructions.

Models fine-tuned on this dataset show improved instruction-following performance.

Enhanced coverage and depth compared to existing datasets.

Abstract

Instruction tuning has become a foundation for unlocking the capabilities of large-scale pretrained models and improving their performance on complex tasks. Thus, the construction of high-quality instruction datasets is crucial for enhancing model performance and generalizability. Although current instruction datasets have reached tens of millions of samples, models finetuned on them may still struggle with complex instruction following and tasks in rare domains. This is primarily due to limited expansion in both ``coverage'' (coverage of task types and knowledge areas) and ``depth'' (instruction complexity) of the instruction set. To address this issue, we propose a systematic instruction data construction framework, which integrates a hierarchical tagging system, an informative seed selection algorithm, an evolutionary data synthesis process, and a model deficiency diagnosis with targeted data generation. These components form an iterative closed-loop to continuously enhance the coverage and depth of instruction data. Based on this framework, we construct Infinity Instruct Subject, a high-quality dataset containing $\sim$1.5 million instructions. Experiments on multiple foundation models and benchmark tasks demonstrate its effectiveness in improving instruction-following capabilities. Further analyses suggest that Infinity Instruct Subject shows enlarged coverage and depth compared to comparable synthesized instruction datasets. Our work lays a theoretical and practical foundation for the efficient, continuous evolution of instruction datasets, moving from data quantity expansion to qualitative improvement.