Understanding the Mechanisms of Fast Hyperparameter Transfer

TL;DR

This paper develops a theoretical framework to understand how hyperparameters can be transferred across different model scales efficiently, especially focusing on the role of problem structure and the Maximal Update Parameterization ($$P).

Contribution

It introduces a formal framework for analyzing hyperparameter transfer, clarifies conditions for fast transfer, and investigates the mechanisms behind $$P's effectiveness in large-scale models.

Findings

Fast transfer is asymptotically more compute-efficient than direct tuning.

Transfer success depends critically on problem structure and parameterization.

Empirical evidence supports the decomposition of optimization trajectories into width-stable and width-sensitive components.

Abstract

The growing scale of deep learning models has rendered standard hyperparameter (HP) optimization prohibitively expensive. A promising solution is the use of scale-aware hyperparameters, which can enable direct transfer of optimal HPs from small-scale grid searches to large models with minimal performance loss. To understand the principles governing such transfer strategy, we develop a general conceptual framework for reasoning about HP transfer across scale, characterizing transfer as fast when the suboptimality it induces vanishes asymptotically faster than the finite-scale performance gap. We show formally that fast transfer is equivalent to useful transfer for compute-optimal grid search, meaning that transfer is asymptotically more compute-efficient than direct tuning. While empirical work has found that the Maximal Update Parameterization ($\mu$P) exhibits fast transfer when scaling model width, the mechanisms remain poorly understood. We show that this property depends critically on problem structure by presenting synthetic settings where transfer either offers provable computational advantage or fails to outperform direct tuning even under $\mu$P. To explain the fast transfer observed in practice, we conjecture that decomposing the optimization trajectory reveals two contributions to loss reduction: (1) a width-stable component that determines the optimal HPs, and (2) a width-sensitive component that improves with width but weakly perturbs the HP optimum. We present empirical evidence for this hypothesis across various settings, including large language model pretraining.