StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Models

TL;DR

StoSignSGD introduces structural stochasticity into SignSGD, fixing its divergence issues on non-smooth objectives and demonstrating superior stability and efficiency in training large language models, especially in low-precision settings.

Contribution

The paper proposes StoSignSGD, a novel unbiased sign-based optimizer that resolves SignSGD's divergence on non-smooth objectives and improves convergence in large language model training.

Findings

StoSignSGD achieves convergence rates matching theoretical lower bounds.

In low-precision FP8 pretraining, StoSignSGD outperforms AdamW and SignSGD in stability and speed.

StoSignSGD improves fine-tuning performance on mathematical reasoning tasks for 7B LLMs.

Abstract

Sign-based optimization algorithms, such as SignSGD, have garnered significant attention for their remarkable performance in distributed learning and training large foundation models. Despite their empirical superiority, SignSGD is known to diverge on non-smooth objectives, which are ubiquitous in modern machine learning due to ReLUs, max-pools, and mixture-of-experts. To overcome this fundamental limitation, we propose \textbf{StoSignSGD}, an algorithm that injects structural stochasticity into the sign operator while maintaining an unbiased update step. In the regime of (online) convex optimization, our theoretical analysis shows that StoSignSGD rigorously resolves the non-convergence issues of SignSGD, achieving a sharp convergence rate matching the lower bound. For the more challenging non-convex non-smooth optimization, we introduce generalized stationary measures that encompass prior definitions, proving that StoSignSGD improves upon the best-known complexity bounds by dimensional factors. Empirically, StoSignSGD exhibits robust stability and superior efficiency across diverse large language model (LLM) training regimes. Notably, in low-precision FP8 pretraining -- a setting where AdamW fails catastrophically -- StoSignSGD remains highly stable and yields a remarkable 1.44$\times$ to 2.14$\times$ speedup relative to established baselines. Furthermore, when fine-tuning 7B LLMs on mathematical reasoning tasks, StoSignSGD delivers substantial performance gains over both AdamW and SignSGD. Finally, to dissect the mechanisms driving its success, we develop a sign conversion framework capable of transforming any general optimizer into its unbiased, sign-based counterpart. Utilizing this framework, we deconstruct the core components of StoSignSGD and present a comprehensive ablation study to empirically validate our algorithmic design choices.