AdamD: Improved bias-correction in Adam

TL;DR

This paper proposes a modification to the Adam optimizer's bias-correction that results in smaller, more appropriate gradient updates during early training, improving stability and hyperparameter sensitivity.

Contribution

It introduces a simplified bias-correction approach focusing only on the second moment estimate, enhancing early training stability in Adam.

Findings

Smaller gradient updates in initial training steps.

Reduced sensitivity to hyperparameters.

Improved early training stability.

Abstract

Here I present a small update to the bias-correction term in the Adam optimizer that has the advantage of making smaller gradient updates in the first several steps of training. With the default bias-correction, Adam may actually make larger than requested gradient updates early in training. By only including the well-justified bias-correction of the second moment gradient estimate, $v_t$, and excluding the bias-correction on the first-order estimate, $m_t$, we attain these more desirable gradient update properties in the first series of steps. The default implementation of Adam may be as sensitive as it is to the hyperparameters $\beta_1, \beta_2$ partially due to the originally proposed bias correction procedure, and its behavior in early steps.