How to Set the Learning Rate for Large-Scale Pre-training?

TL;DR

This paper investigates how to accurately set the learning rate for large-scale pre-training, proposing new scaling laws and transfer methods, and analyzing their effectiveness and limitations through extensive experiments.

Contribution

It introduces a novel Scaling Law for search factor, extends $$Transfer to MoE architectures, and provides a comprehensive comparison and analysis of hyperparameter tuning paradigms in large-scale pre-training.

Findings

Scaling Law reduces search complexity from O(n^3) to O(n*C_D*C_{ta})

Transfer paradigm's scalability is limited in large-scale scenarios

Module-wise parameter tuning underperforms in large-scale pre-training

Abstract

Optimal configuration of the learning rate (LR) is a fundamental yet formidable challenge in large-scale pre-training. Given the stringent trade-off between training costs and model performance, the pivotal question is whether the optimal LR can be accurately extrapolated from low-cost experiments. In this paper, we formalize this investigation into two distinct research paradigms: Fitting and Transfer. Within the Fitting Paradigm, we innovatively introduce a Scaling Law for search factor, effectively reducing the search complexity from O(n^3) to O(n*C_D*C_{\eta}) via predictive modeling. Within the Transfer Paradigm, we extend the principles of $\mu$Transfer to the Mixture of Experts (MoE) architecture, broadening its applicability to encompass model depth, weight decay, and token horizons. By pushing the boundaries of existing hyperparameter research in terms of scale, we conduct a comprehensive comparison between these two paradigms. Our empirical results challenge the scalability of the widely adopted $\mu$ Transfer in large-scale pre-training scenarios. Furthermore, we provide a rigorous analysis through the dual lenses of training stability and feature learning to elucidate the underlying reasons why module-wise parameter tuning underperforms in large-scale settings. This work offers systematic practical guidelines and a fresh theoretical perspective for optimizing industrial-level pre-training.