The Epochal Sawtooth Phenomenon: Unveiling Training Loss Oscillations in Adam and Other Optimizers

TL;DR

This paper uncovers and analyzes the Epochal Sawtooth Phenomenon (ESP), a recurring loss pattern during training with Adam and similar optimizers, caused by adaptive learning rate adjustments and data shuffling effects.

Contribution

It is the first comprehensive analysis of ESP, linking it to optimizer parameters, data shuffling, and model capacity, supported by empirical and simplified quadratic minimization experiments.

Findings

ESP is most pronounced with Adam optimizer.

Smaller $eta_2$ values increase ESP severity.

ESP can occur in simple quadratic optimization tasks.

Abstract

In this paper, we identify and analyze a recurring training loss pattern, which we term the \textit{Epochal Sawtooth Phenomenon (ESP)}, commonly observed during training with adaptive gradient-based optimizers, particularly Adam optimizer. This pattern is characterized by a sharp drop in loss at the beginning of each epoch, followed by a gradual increase, resulting in a sawtooth-shaped loss curve. Through empirical observations, we demonstrate that while this effect is most pronounced with Adam, it persists, although less severely, with other optimizers such as RMSProp. We empirically analyze the mechanisms underlying ESP, focusing on key factors such as Adam's $\beta$ parameters, batch size, data shuffling, and sample replacement. Our analysis shows that ESP arises from adaptive learning rate adjustments controlled by the second moment estimate. Additionally, we identify the ``immediate re-exposure to samples'' effect during data shuffling, which causes the model to learn or memorize more at the beginning of each epoch. We also find that smaller values of $\beta_2$ exacerbate ESP but can act as a form of regularization. While ESP is not necessarily indicative of overfitting, higher model capacity can amplify the phenomenon. To further support our analysis, we replicate ESP through a high-dimensional quadratic minimization task. We demonstrate that ESP can emerge even in simple optimization scenarios, reinforcing the generality of this pattern. The code for reproducing our experiments is available at https://github.com/qiliuchn/training-loss-pattern.