Stochastic Weakly Convex Optimization Under Heavy-Tailed Noises

TL;DR

This paper studies the convergence behavior of stochastic weakly convex optimization algorithms under heavy-tailed noises, providing new theoretical insights into their performance in non-convex, non-smooth settings.

Contribution

It extends convergence analysis of stochastic subgradient methods to weakly convex functions under heavy-tailed noises, including sub-Weibull and p-BCM assumptions.

Findings

Vanilla SsGD's dependence on failure probability remains stable under sub-Weibull noise.

Clipped SsGD's dependence on failure probability is unaffected by non-convexity and non-smoothness under p-BCM noise.

Sample complexity under p-BCM noise is worse than the lower bound for smooth optimization.

Abstract

An increasing number of studies have focused on stochastic first-order methods (SFOMs) under heavy-tailed gradient noises, which have been observed in the training of practical deep learning models. In this paper, we focus on two types of gradient noises: one is sub-Weibull noise, and the other is noise under the assumption that it has a bounded $p$-th central moment ($p$-BCM) with $p\in (1, 2]$. The latter is more challenging due to the occurrence of infinite variance when $p\in (1, 2)$. Under these two gradient noise assumptions, the in-expectation and high-probability convergence of SFOMs have been extensively studied in the contexts of convex optimization and standard smooth optimization. However, for weakly convex objectives-a class that includes all Lipschitz-continuous convex objectives and smooth objectives-our understanding of the in-expectation and high-probability convergence of SFOMs under these two types of noises remains incomplete. We investigate the high-probability convergence of the vanilla stochastic subgradient descent (SsGD) method under sub-Weibull noises, as well as the high-probability and in-expectation convergence of clipped SsGD under the $p$-BCM noises. Both analyses are conducted in the context of weakly convex optimization. For weakly convex objectives that may be non-convex and non-smooth, our results demonstrate that the theoretical dependence of vanilla SsGD on the failure probability and number of iterations under sub-Weibull noises does not degrade compared to the case of smooth objectives. Under $p$-BCM noises, our findings indicate that the non-smoothness and non-convexity of weakly convex objectives do not impact the theoretical dependence of clipped SGD on the failure probability relative to the smooth case; however, the sample complexity we derived is worse than a well-known lower bound for smooth optimization.