Non-Asymptotic Analysis of Online Multiplicative Stochastic Gradient Descent

TL;DR

This paper provides a non-asymptotic analysis of the multiplicative stochastic gradient descent (M-SGD), demonstrating universality in noise behavior and showing that its error distribution approximates a scaled Gaussian.

Contribution

It extends the understanding of SGD by analyzing a more general multiplicative noise model and establishing non-asymptotic bounds and distributional approximations.

Findings

Noise classes with same mean and covariance have similar properties.

Non-asymptotic Wasserstein bounds for M-SGD.

M-SGD error approximates a scaled Gaussian distribution.

Abstract

Past research has indicated that the covariance of the Stochastic Gradient Descent (SGD) error done via minibatching plays a critical role in determining its regularization and escape from low potential points. Motivated by some new research in this area, we prove universality results by showing that noise classes that have the same mean and covariance structure of SGD via minibatching have similar properties. We mainly consider the Multiplicative Stochastic Gradient Descent (M-SGD) algorithm as introduced in previous work, which has a much more general noise class than the SGD algorithm done via minibatching. We establish non asymptotic bounds for the M-SGD algorithm in the Wasserstein distance. We also show that the M-SGD error is approximately a scaled Gaussian distribution with mean $0$ at any fixed point of the M-SGD algorithm.