OpenCSG Chinese Corpus: A Series of High-quality Chinese Datasets for LLM Training

TL;DR

The paper introduces the OpenCSG Chinese Corpus, a collection of high-quality Chinese datasets designed to enhance the training and performance of large language models across various tasks.

Contribution

It presents a diverse, high-quality Chinese dataset series with scalable curation processes, specifically tailored for LLM pretraining, post-training, and fine-tuning, addressing data scarcity issues.

Findings

Significant performance improvements on Chinese NLP tasks.

Effective training of smaller parameter models.

Demonstrated corpus versatility across domains.

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities, but their success heavily relies on the quality of pretraining corpora. For Chinese LLMs, the scarcity of high-quality Chinese datasets presents a significant challenge, often limiting their performance. To address this issue, we propose the OpenCSG Chinese Corpus, a series of high-quality datasets specifically designed for LLM pretraining, post-training, and fine-tuning. This corpus includes Fineweb-edu-chinese, Fineweb-edu-chinese-v2, Cosmopedia-chinese, and Smoltalk-chinese, each with distinct characteristics: Fineweb-edu datasets focus on filtered, high-quality content derived from diverse Chinese web sources; Cosmopedia-chinese provides synthetic, textbook-style data for knowledge-intensive training; and Smoltalk-chinese emphasizes stylistic and diverse chat-format data. The OpenCSG Chinese Corpus is characterized by its high-quality text, diverse coverage across domains, and scalable, reproducible data curation processes. Additionally, we conducted extensive experimental analyses, including evaluations on smaller parameter models, which demonstrated significant performance improvements in tasks such as C-Eval, showcasing the effectiveness of the corpus for training Chinese LLMs.