Do We Need Adam? Surprisingly Strong and Sparse Reinforcement Learning with SGD in LLMs

TL;DR

This paper shows that stochastic gradient descent (SGD) can match or outperform AdamW in reinforcement learning for large language models, achieving high parameter efficiency and challenging common optimization practices.

Contribution

It demonstrates that SGD, a simpler optimizer, is more effective and memory-efficient than AdamW for RL in LLMs, with significant parameter sparsity in updates.

Findings

SGD matches or outperforms AdamW in RL training of LLMs.

Full fine-tuning with SGD updates fewer than 0.02% of parameters.

RL training benefits less from Adam's adaptive features than supervised learning.

Abstract

Reinforcement learning (RL), particularly RL from verifiable reward (RLVR), has become a crucial phase of training large language models (LLMs) and a key focus of current scaling efforts. However, optimization practices in RL largely follow those of next-token prediction stages (e.g., pretraining and supervised fine-tuning), despite fundamental differences between RL and these stages highlighted by recent work. One such practice is the use of the AdamW optimizer, which is widely adopted for training large-scale transformers despite its high memory overhead. Our analysis shows that both momentum and adaptive learning rates in AdamW are less influential in RL than in SFT, leading us to hypothesize that RL benefits less from Adam-style per-parameter adaptive learning rates and momentum. Confirming this hypothesis, our experiments demonstrate that the substantially more memory-efficient SGD, which is known to perform poorly in supervised learning of large-scale transformers, matches or even outperforms AdamW in RL for LLMs. Remarkably, full fine-tuning with SGD updates fewer than 0.02% of model parameters without any sparsity-promoting regularization, more than 1000 times fewer than AdamW. Our analysis offers potential reasons for this update sparsity. These findings provide new insights into the optimization dynamics of RL in LLMs and show that RL can be substantially more parameter-efficient than previously recognized.