D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models

TL;DR

This paper introduces the D-CPT Law, a scaling law that predicts optimal domain-specific continual pre-training configurations for large language models, reducing costly trial-and-error and enabling efficient domain adaptation.

Contribution

The paper proposes the D-CPT Law and Cross-Domain D-CPT Law, enabling accurate prediction of optimal mixture ratios and performance for domain-specific LLM pre-training with minimal training costs.

Findings

D-CPT Law accurately predicts performance across various domains.

Cross-Domain D-CPT Law requires only 1% of training costs for target domain prediction.

Experimental results validate the effectiveness and generalizability of the proposed laws.

Abstract

Continual Pre-Training (CPT) on Large Language Models (LLMs) has been widely used to expand the model's fundamental understanding of specific downstream domains (e.g., math and code). For the CPT on domain-specific LLMs, one important question is how to choose the optimal mixture ratio between the general-corpus (e.g., Dolma, Slim-pajama) and the downstream domain-corpus. Existing methods usually adopt laborious human efforts by grid-searching on a set of mixture ratios, which require high GPU training consumption costs. Besides, we cannot guarantee the selected ratio is optimal for the specific domain. To address the limitations of existing methods, inspired by the Scaling Law for performance prediction, we propose to investigate the Scaling Law of the Domain-specific Continual Pre-Training (D-CPT Law) to decide the optimal mixture ratio with acceptable training costs for LLMs of different sizes. Specifically, by fitting the D-CPT Law, we can easily predict the general and downstream performance of arbitrary mixture ratios, model sizes, and dataset sizes using small-scale training costs on limited experiments. Moreover, we also extend our standard D-CPT Law on cross-domain settings and propose the Cross-Domain D-CPT Law to predict the D-CPT law of target domains, where very small training costs (about 1% of the normal training costs) are needed for the target domains. Comprehensive experimental results on six downstream domains demonstrate the effectiveness and generalizability of our proposed D-CPT Law and Cross-Domain D-CPT Law.