A Practice of Post-Training on Llama-3 70B with Optimal Selection of Additional Language Mixture Ratio

TL;DR

This paper systematically studies how to optimize the mixture ratio of additional language data during continual pre-training of Llama-3 models, improving their performance in Chinese and specific domains.

Contribution

It provides a detailed analysis of the optimal correlation between language mixture ratio and learning rate, enhancing Llama-3's capabilities in Chinese, math, coding, and emotional intelligence.

Findings

Optimal hyper-parameter choices improve Chinese language skills.

Fine-tuning enhances performance in math, coding, and emotional intelligence.

Deployed 70B model performs well in real-life chat systems.

Abstract

Large Language Models (LLM) often need to be Continual Pre-Trained (CPT) to obtain unfamiliar language skills or adapt to new domains. The huge training cost of CPT often asks for cautious choice of key hyper-parameters such as the mixture ratio of extra language or domain corpus. However, there is no systematic study that bridges the gap between the optimal mixture ratio and the actual model performance, and the gap between experimental scaling law and the actual deployment in the full model size. In this paper, we perform CPT on Llama-3 8B and 70B to enhance its Chinese ability. We study the optimal correlation between the Additional Language Mixture Ratio (ALMR) and the Learning Rate (LR) on the 8B size which directly indicates the optimal experimental setup. By thorough choice of hyper-parameter, and subsequent fine-tuning, the model capability is improved not only on the Chinese-related benchmark but also in some specific domains including math, coding, and emotional intelligence. We deploy the final 70B version of LLM on a real-life chat system which obtains satisfying performance.