Understanding AdamW through Proximal Methods and Scale-Freeness

TL;DR

This paper explains the advantages of AdamW over Adam-$\\ell_2$ by interpreting it as a proximal gradient method and exploring its scale-free property, supported by empirical evidence across various deep learning tasks.

Contribution

It provides a novel interpretation of AdamW as a proximal method and links its scale-invariance to improved generalization in deep learning.

Findings

AdamW can be viewed as an approximation of a proximal gradient method.

AdamW's scale-freeness correlates with its performance advantages.

Empirical results across multiple tasks support the hypothesis about scale-invariance benefits.

Abstract

Adam has been widely adopted for training deep neural networks due to less hyperparameter tuning and remarkable performance. To improve generalization, Adam is typically used in tandem with a squared $\ell_2$ regularizer (referred to as Adam-$\ell_2$). However, even better performance can be obtained with AdamW, which decouples the gradient of the regularizer from the update rule of Adam-$\ell_2$. Yet, we are still lacking a complete explanation of the advantages of AdamW. In this paper, we tackle this question from both an optimization and an empirical point of view. First, we show how to re-interpret AdamW as an approximation of a proximal gradient method, which takes advantage of the closed-form proximal mapping of the regularizer instead of only utilizing its gradient information as in Adam-$\ell_2$. Next, we consider the property of "scale-freeness" enjoyed by AdamW and by its proximal counterpart: their updates are invariant to component-wise rescaling of the gradients. We provide empirical evidence across a wide range of deep learning experiments showing a correlation between the problems in which AdamW exhibits an advantage over Adam-$\ell_2$ and the degree to which we expect the gradients of the network to exhibit multiple scales, thus motivating the hypothesis that the advantage of AdamW could be due to the scale-free updates.