AdaFamily: A family of Adam-like adaptive gradient methods

TL;DR

AdaFamily introduces a new family of adaptive gradient methods that blend features of Adam, AdaBelief, and AdaMomentum, demonstrating improved performance in deep neural network training.

Contribution

It presents a novel family of adaptive gradient algorithms that unify and extend existing methods like Adam, AdaBelief, and AdaMomentum.

Findings

Outperforms existing algorithms on standard image classification datasets

Demonstrates improved training stability and accuracy

Validates effectiveness across multiple neural network architectures

Abstract

We propose AdaFamily, a novel method for training deep neural networks. It is a family of adaptive gradient methods and can be interpreted as sort of a blend of the optimization algorithms Adam, AdaBelief and AdaMomentum. We perform experiments on standard datasets for image classification, demonstrating that our proposed method outperforms these algorithms.