Understanding Short-Range Memory Effects in Deep Neural Networks

TL;DR

This paper proposes that stochastic gradient noise in SGD behaves like fractional Brownian motion, explaining its tendency to favor flat minima and providing new insights into its convergence and generalization properties.

Contribution

The study introduces a novel perspective that models SGD as a discretization of an SDE driven by fractional Brownian motion, highlighting the role of short-range memory effects.

Findings

SGN is neither Gaussian nor Levy stable.

SGD favors flat minima with longer residence times.

Short-range memory effects are consistent across models and datasets.

Abstract

Stochastic gradient descent (SGD) is of fundamental importance in deep learning. Despite its simplicity, elucidating its efficacy remains challenging. Conventionally, the success of SGD is ascribed to the stochastic gradient noise (SGN) incurred in the training process. Based on this consensus, SGD is frequently treated and analyzed as the Euler-Maruyama discretization of stochastic differential equations (SDEs) driven by either Brownian or Levy stable motion. In this study, we argue that SGN is neither Gaussian nor Levy stable. Instead, inspired by the short-range correlation emerging in the SGN series, we propose that SGD can be viewed as a discretization of an SDE driven by fractional Brownian motion (FBM). Accordingly, the different convergence behavior of SGD dynamics is well-grounded. Moreover, the first passage time of an SDE driven by FBM is approximately derived. The result suggests a lower escaping rate for a larger Hurst parameter, and thus SGD stays longer in flat minima. This happens to coincide with the well-known phenomenon that SGD favors flat minima that generalize well. Extensive experiments are conducted to validate our conjecture, and it is demonstrated that short-range memory effects persist across various model architectures, datasets, and training strategies. Our study opens up a new perspective and may contribute to a better understanding of SGD.