AdaPlus: Integrating Nesterov Momentum and Precise Stepsize Adjustment on AdamW Basis

TL;DR

AdaPlus is a new optimizer that combines Nesterov momentum and precise stepsize adjustment on AdamW, outperforming other optimizers in various machine learning tasks without adding extra hyperparameters.

Contribution

It introduces AdaPlus, an optimizer integrating Nesterov momentum and stepsize adjustment on AdamW, with no additional hyperparameters, validated through extensive experiments.

Findings

AdaPlus performs comparably or better than SGD with momentum on image classification.

AdaPlus outperforms other state-of-the-art optimizers on language modeling.

AdaPlus demonstrates high stability in training GANs.

Abstract

This paper proposes an efficient optimizer called AdaPlus which integrates Nesterov momentum and precise stepsize adjustment on AdamW basis. AdaPlus combines the advantages of AdamW, Nadam, and AdaBelief and, in particular, does not introduce any extra hyper-parameters. We perform extensive experimental evaluations on three machine learning tasks to validate the effectiveness of AdaPlus. The experiment results validate that AdaPlus (i) among all the evaluated adaptive methods, performs most comparable with (even slightly better than) SGD with momentum on image classification tasks and (ii) outperforms other state-of-the-art optimizers on language modeling tasks and illustrates pretty high stability when training GANs. The experiment code of AdaPlus will be accessible at: https://github.com/guanleics/AdaPlus.