CAME: Confidence-guided Adaptive Memory Efficient Optimization

TL;DR

CAME is a new optimizer that combines fast convergence and low memory usage for training large language models, using a confidence-guided strategy to improve stability and performance.

Contribution

We introduce CAME, a memory-efficient adaptive optimizer with confidence-guided stabilization, achieving superior training speed and accuracy over existing methods.

Findings

CAME outperforms Adam in BERT pre-training with large batch sizes.

CAME demonstrates faster convergence and higher accuracy across NLP tasks.

The method maintains training stability with reduced memory overhead.

Abstract

Adaptive gradient methods, such as Adam and LAMB, have demonstrated excellent performance in the training of large language models. Nevertheless, the need for adaptivity requires maintaining second-moment estimates of the per-parameter gradients, which entails a high cost of extra memory overheads. To solve this problem, several memory-efficient optimizers (e.g., Adafactor) have been proposed to obtain a drastic reduction in auxiliary memory usage, but with a performance penalty. In this paper, we first study a confidence-guided strategy to reduce the instability of existing memory efficient optimizers. Based on this strategy, we propose CAME to simultaneously achieve two goals: fast convergence as in traditional adaptive methods, and low memory usage as in memory-efficient methods. Extensive experiments demonstrate the training stability and superior performance of CAME across various NLP tasks such as BERT and GPT-2 training. Notably, for BERT pre-training on the large batch size of 32,768, our proposed optimizer attains faster convergence and higher accuracy compared with the Adam optimizer. The implementation of CAME is publicly available.