EXAdam: The Power of Adaptive Cross-Moments

TL;DR

EXAdam is an enhanced optimization algorithm building on Adam, introducing new debiasing and gradient acceleration techniques to improve convergence, robustness, and effectiveness in complex landscapes, demonstrated by faster training and better accuracy on CIFAR-10.

Contribution

The paper proposes EXAdam, a novel optimizer with debiasing and acceleration mechanisms, offering theoretical insights and empirical improvements over Adam.

Findings

38.46% faster convergence on CIFAR-10

Improved training, validation, and testing accuracies

Potential robustness to hyperparameters

Abstract

This paper introduces EXAdam ($\textbf{EX}$tended $\textbf{Adam}$), a novel optimization algorithm that builds upon the widely-used Adam optimizer. EXAdam incorporates two key enhancements: (1) new debiasing terms for improved moment estimation and (2) a gradient-based acceleration mechanism for increased responsiveness to the current loss landscape. These innovations work synergistically to address limitations of the original Adam algorithm, potentially offering improved convergence properties, enhanced ability to escape saddle points, and potentially greater robustness to hyperparameter choices, though this requires further investigation. We provide a theoretical analysis of EXAdam's components and their interactions, highlighting the algorithm's potential advantages in navigating complex optimization landscapes. Empirical evaluations demonstrate EXAdam's superiority over Adam, achieving 38.46% faster convergence and yielding improvements of 1.96%, 2.17%, and 1.17% in training, validation, and testing accuracies, respectively, when applied to a CNN trained on the CIFAR-10 dataset. While these results are promising, further empirical validation across diverse tasks is essential to fully gauge EXAdam's efficacy. Nevertheless, EXAdam represents a significant advancement in adaptive optimization techniques, with promising implications for a wide range of machine learning applications. This work aims to contribute to the ongoing development of more efficient, adaptive, and universally applicable optimization methods in the field of machine learning and artificial intelligence.