A DNN Optimizer that Improves over AdaBelief by Suppression of the Adaptive Stepsize Range

TL;DR

This paper introduces Aida, an improved optimizer based on AdaBelief that suppresses the adaptive stepsize range, leading to better performance across various neural network training tasks.

Contribution

The paper proposes a novel optimizer, Aida, which extends AdaBelief by suppressing adaptive stepsize range through layerwise vector projections, improving training outcomes.

Findings

Aida outperforms nine optimizers on NLP and image classification tasks.

Aida matches top performance on image generation models.

Aida yields higher validation accuracy than AdaBelief on ImageNet.

Abstract

We make contributions towards improving adaptive-optimizer performance. Our improvements are based on suppression of the range of adaptive stepsizes in the AdaBelief optimizer. Firstly, we show that the particular placement of the parameter epsilon within the update expressions of AdaBelief reduces the range of the adaptive stepsizes, making AdaBelief closer to SGD with momentum. Secondly, we extend AdaBelief by further suppressing the range of the adaptive stepsizes. To achieve the above goal, we perform mutual layerwise vector projections between the gradient g_t and its first momentum m_t before using them to estimate the second momentum. The new optimization method is referred to as Aida. Thirdly, extensive experimental results show that Aida outperforms nine optimizers when training transformers and LSTMs for NLP, and VGG and ResNet for image classification over CIAF10 and CIFAR100 while matching the best performance of the nine methods when training WGAN-GP models for image generation tasks. Furthermore, Aida produces higher validation accuracies than AdaBelief for training ResNet18 over ImageNet. Code is available <a href="https://github.com/guoqiang-x-zhang/AidaOptimizer">at this URL</a>