Investigating Alternatives to the Root Mean Square for Adaptive Gradient Methods

TL;DR

This paper explores alternative $L^p$ normalization norms in adaptive gradient methods like Adam, showing that using $p > 2$ can enhance learning speed and performance across benchmarks.

Contribution

It provides the first theoretical and empirical analysis of how different $L^p$ norms affect adaptive gradient methods, demonstrating improvements with $p > 2$.

Findings

$p > 2$ improves learning speed and final performance

$p=3$ or $p=6$ outperform state-of-the-art methods

The choice of $p$ influences step size without affecting other properties

Abstract

Adam is an adaptive gradient method that has experienced widespread adoption due to its fast and reliable training performance. Recent approaches have not offered significant improvement over Adam, often because they do not innovate upon one of its core features: normalization by the root mean square (RMS) of recent gradients. However, as noted by Kingma and Ba (2015), any number of $L^p$ normalizations are possible, with the RMS corresponding to the specific case of $p=2$. In our work, we theoretically and empirically characterize the influence of different $L^p$ norms on adaptive gradient methods for the first time. We show mathematically how the choice of $p$ influences the size of the steps taken, while leaving other desirable properties unaffected. We evaluate Adam with various $L^p$ norms on a suite of deep learning benchmarks, and find that $p > 2$ consistently leads to improved learning speed and final performance. The choices of $p=3$ or $p=6$ also match or outperform state-of-the-art methods in all of our experiments.