Can Adaptive Gradient Methods Converge under Heavy-Tailed Noise? A Case Study of AdaGrad

TL;DR

This paper investigates whether adaptive gradient methods like AdaGrad can converge under heavy-tailed gradient noise without modifications, providing new theoretical convergence rates and bounds in non-convex optimization.

Contribution

It offers the first provable convergence rate for AdaGrad under heavy-tailed noise, without prior knowledge of the tail index, and introduces bounds showing limitations of AdaGrad compared to minimax rates.

Findings

AdaGrad converges under heavy-tailed noise with a rate depending on the tail index p.

An algorithm-dependent lower bound indicates AdaGrad cannot attain the minimax rate.

AdaGrad-Norm achieves an improved rate under mild additional assumptions.

Abstract

Many tasks in modern machine learning are observed to involve heavy-tailed gradient noise during the optimization process. To manage this realistic and challenging setting, new mechanisms, such as gradient clipping and gradient normalization, have been introduced to ensure the convergence of first-order algorithms. However, adaptive gradient methods, a famous class of modern optimizers that includes popular $\mathtt{Adam}$ and $\mathtt{AdamW}$, often perform well even without any extra operations mentioned above. It is therefore natural to ask whether adaptive gradient methods can converge under heavy-tailed noise without any algorithmic changes. In this work, we take the first step toward answering this question by investigating a special case, $\mathtt{AdaGrad}$, the origin of adaptive gradient methods. We provide the first provable convergence rate for $\mathtt{AdaGrad}$ in non-convex optimization when the tail index $p$ satisfies $4/3<p\leq2$. Notably, this result is achieved without requiring any prior knowledge of $p$ and is hence adaptive to the tail index. In addition, we develop an algorithm-dependent lower bound, suggesting that the existing minimax rate for heavy-tailed optimization is not attainable by $\mathtt{AdaGrad}$. Lastly, we consider $\mathtt{AdaGrad}\text{-}\mathtt{Norm}$, a popular variant of $\mathtt{AdaGrad}$ in theoretical studies, and show an improved rate that holds for any $1<p\leq2$ under an extra mild assumption.