An Empirical Study of Instruction-tuning Large Language Models in Chinese

TL;DR

This paper provides an empirical analysis of instruction-tuning large language models in Chinese, exploring key factors affecting performance and aiming to enhance Chinese LLMs like ChatGPT.

Contribution

It systematically investigates the effects of base models, parameter-efficient methods, and data types on instruction-tuning Chinese LLMs, offering practical insights.

Findings

Instruction data type significantly impacts tuning effectiveness.

Parameter-efficient methods can achieve competitive performance.

Additional factors like chain-of-thought data influence results.

Abstract

The success of ChatGPT validates the potential of large language models (LLMs) in artificial general intelligence (AGI). Subsequently, the release of LLMs has sparked the open-source community's interest in instruction-tuning, which is deemed to accelerate ChatGPT's replication process. However, research on instruction-tuning LLMs in Chinese, the world's most spoken language, is still in its early stages. Therefore, this paper makes an in-depth empirical study of instruction-tuning LLMs in Chinese, which can serve as a cookbook that provides valuable findings for effectively customizing LLMs that can better respond to Chinese instructions. Specifically, we systematically explore the impact of LLM bases, parameter-efficient methods, instruction data types, which are the three most important elements for instruction-tuning. Besides, we also conduct experiment to study the impact of other factors, e.g., chain-of-thought data and human-value alignment. We hope that this empirical study can make a modest contribution to the open Chinese version of ChatGPT. This paper will release a powerful Chinese LLMs that is comparable to ChatGLM. The code and data are available at https://github.com/PhoebusSi/Alpaca-CoT.