Dynamics of Instruction Fine-Tuning for Chinese Large Language Models

TL;DR

This paper systematically studies how data quantity, model size, and data construction influence instruction tuning of Chinese LLMs, revealing ability-specific scaling behaviors and strategies for efficient training.

Contribution

It introduces a comprehensive analysis of instruction tuning effects on Chinese LLMs, highlighting ability-specific scaling sensitivities and tailored training strategies.

Findings

Some abilities are more responsive to scaling than others.

Scaling sensitivity is explained by Complexity and Transference features.

Tailored training strategies improve performance on benchmarks.

Abstract

Instruction tuning is a burgeoning method to elicit the general intelligence of Large Language Models (LLMs). While numerous studies have examined the impact of factors such as data volume and model size on English models, the scaling properties of instruction tuning in other languages remain largely unexplored. In this work, we systematically investigate the effects of data quantity, model size, and data construction methods on instruction tuning for Chinese LLMs. We utilize a newly curated dataset, DoIT, which includes over 40,000 high-quality instruction instances covering ten underlying abilities, such as creative writing, code generation, and logical reasoning. Our experiments, conducted on models ranging from 7b to 33b parameters, yield three key findings: (i) While these factors directly affect overall model performance, some abilities are more responsive to scaling, whereas others demonstrate significant resistance. (ii) The scaling sensitivity of different abilities to these factors can be explained by two features: Complexity and Transference. (iii) By tailoring training strategies to their varying sensitivities, specific abilities can be efficiently learned, enhancing performance on two public benchmarks.