Revealing Modular Gradient Noise Imbalance in LLMs: Calibrating Adam via Signal-to-Noise Ratio

TL;DR

This paper introduces MoLS, a method that uses signal-to-noise ratio estimates to automatically scale learning rates across modules in LLMs, improving training efficiency and performance.

Contribution

The paper proposes MoLS, a novel module-wise learning rate scaling method based on SNR, addressing heterogeneity in LLM training without manual tuning.

Findings

MoLS improves convergence speed in LLM training benchmarks.

MoLS achieves performance comparable to manual module-specific learning rates.

MoLS is compatible with memory-efficient training algorithms.

Abstract

The impressive performance of large language models (LLMs) arises from their massive scale and heterogeneous module composition. However, this structural heterogeneity introduces additional optimization challenges. While adaptive optimizers such as Adam(W) provide per-parameter adaptivity, they do not explicitly account for module-level gradient heterogeneity, resulting in slower convergence, suboptimal performance, or training instability. Existing approaches typically rely on manually tuned module-specific learning rates or specific optimization strategies, which are computationally costly and difficult to generalize across tasks or models. To establish a more principled approach, we first analyze the noise-damping behavior of Adam in high-noise modules and introduce \textbf{Module-wise Learning Rate Scaling via SNR (MoLS)}. MoLS estimates module-level SNRs to scale Adam updates, allowing automated module-wise learning rate allocation without manual tuning. Empirical results through multiple LLM training benchmarks demonstrate that MoLS improves convergence speed and generalization, achieving performance comparable to carefully tuned module-specific learning rates, while remaining compatible with memory-efficient training algorithms.