On Higher-order Moments in Adam

TL;DR

This paper introduces HAdam, an extension of Adam that incorporates higher-order moments of stochastic gradients, leading to improved performance in deep learning optimization.

Contribution

We propose HAdam, a novel optimizer utilizing higher-order moments of stochastic gradients, and analyze its empirical and theoretical advantages over Adam.

Findings

Higher-order moments improve optimization performance.

HAdam outperforms Adam in experiments.

Odd and even moments have distinct effects.

Abstract

In this paper, we investigate the popular deep learning optimization routine, Adam, from the perspective of statistical moments. While Adam is an adaptive lower-order moment based (of the stochastic gradient) method, we propose an extension namely, HAdam, which uses higher order moments of the stochastic gradient. Our analysis and experiments reveal that certain higher-order moments of the stochastic gradient are able to achieve better performance compared to the vanilla Adam algorithm. We also provide some analysis of HAdam related to odd and even moments to explain some intriguing and seemingly non-intuitive empirical results.