On the Theoretical Properties of Noise Correlation in Stochastic Optimization

TL;DR

This paper investigates the properties of correlated noise in stochastic optimization, introducing a new algorithm based on fractional Brownian motion that enhances exploration capabilities in non-convex landscapes.

Contribution

It introduces fPGD, a novel algorithm utilizing correlated noise from fractional Brownian motion, and analyzes its theoretical and empirical advantages over existing methods.

Findings

fPGD exhibits improved exploration in non-convex landscapes.

Correlated noise can outperform traditional uncorrelated noise in certain scenarios.

Theoretical analysis supports the empirical benefits of fPGD.

Abstract

Studying the properties of stochastic noise to optimize complex non-convex functions has been an active area of research in the field of machine learning. Prior work has shown that the noise of stochastic gradient descent improves optimization by overcoming undesirable obstacles in the landscape. Moreover, injecting artificial Gaussian noise has become a popular idea to quickly escape saddle points. Indeed, in the absence of reliable gradient information, the noise is used to explore the landscape, but it is unclear what type of noise is optimal in terms of exploration ability. In order to narrow this gap in our knowledge, we study a general type of continuous-time non-Markovian process, based on fractional Brownian motion, that allows for the increments of the process to be correlated. This generalizes processes based on Brownian motion, such as the Ornstein-Uhlenbeck process. We demonstrate how to discretize such processes which gives rise to the new algorithm fPGD. This method is a generalization of the known algorithms PGD and Anti-PGD. We study the properties of fPGD both theoretically and empirically, demonstrating that it possesses exploration abilities that, in some cases, are favorable over PGD and Anti-PGD. These results open the field to novel ways to exploit noise for training machine learning models.