Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model

TL;DR

This paper introduces CT-LLM, a 2-billion-parameter Chinese-centric large language model trained primarily on Chinese data, demonstrating superior Chinese language understanding and performance, and open-sourcing the training process and benchmarks.

Contribution

The study presents a novel Chinese-centric LLM trained from scratch with an extensive Chinese corpus, diverging from traditional English-focused training approaches.

Findings

Achieves high performance on Chinese language tasks

Demonstrates proficiency in English through supervised fine-tuning

Open-sources training data, process, and benchmarks

Abstract

In this study, we introduce CT-LLM, a 2B large language model (LLM) that illustrates a pivotal shift towards prioritizing the Chinese language in developing LLMs. Uniquely initiated from scratch, CT-LLM diverges from the conventional methodology by primarily incorporating Chinese textual data, utilizing an extensive corpus of 1,200 billion tokens, including 800 billion Chinese tokens, 300 billion English tokens, and 100 billion code tokens. This strategic composition facilitates the model's exceptional proficiency in understanding and processing Chinese, a capability further enhanced through alignment techniques. Demonstrating remarkable performance on the CHC-Bench, CT-LLM excels in Chinese language tasks, and showcases its adeptness in English through SFT. This research challenges the prevailing paradigm of training LLMs predominantly on English corpora and then adapting them to other languages, broadening the horizons for LLM training methodologies. By open-sourcing the full process of training a Chinese LLM, including a detailed data processing procedure with the obtained Massive Appropriate Pretraining Chinese Corpus (MAP-CC), a well-chosen multidisciplinary Chinese Hard Case Benchmark (CHC-Bench), and the 2B-size Chinese Tiny LLM (CT-LLM), we aim to foster further exploration and innovation in both academia and industry, paving the way for more inclusive and versatile language models.