Practical Sharpness-Aware Minimization Cannot Converge All the Way to Optima

TL;DR

This paper investigates the convergence properties of practical Sharpness-Aware Minimization (SAM) algorithms with constant perturbation and gradient normalization, revealing limitations in reaching global optima and highlighting differences from theoretical assumptions.

Contribution

It provides the first analysis of practical SAM configurations, demonstrating their limited convergence to optima and the unavoidable neighborhood bounds caused by fixed perturbation size.

Findings

Deterministic SAM achieves rac{1}{T^2} convergence rate for strongly convex functions.

Stochastic SAM converges only up to an O( ho^2) neighborhood of the optimum.

Examples show the O( ho^2) bounds are unavoidable in practical SAM settings.

Abstract

Sharpness-Aware Minimization (SAM) is an optimizer that takes a descent step based on the gradient at a perturbation $y_t = x_t + \rho \frac{\nabla f(x_t)}{\lVert \nabla f(x_t) \rVert}$ of the current point $x_t$. Existing studies prove convergence of SAM for smooth functions, but they do so by assuming decaying perturbation size $\rho$ and/or no gradient normalization in $y_t$, which is detached from practice. To address this gap, we study deterministic/stochastic versions of SAM with practical configurations (i.e., constant $\rho$ and gradient normalization in $y_t$) and explore their convergence properties on smooth functions with (non)convexity assumptions. Perhaps surprisingly, in many scenarios, we find out that SAM has limited capability to converge to global minima or stationary points. For smooth strongly convex functions, we show that while deterministic SAM enjoys tight global convergence rates of $\tilde \Theta(\frac{1}{T^2})$, the convergence bound of stochastic SAM suffers an inevitable additive term $O(\rho^2)$, indicating convergence only up to neighborhoods of optima. In fact, such $O(\rho^2)$ factors arise for stochastic SAM in all the settings we consider, and also for deterministic SAM in nonconvex cases; importantly, we prove by examples that such terms are unavoidable. Our results highlight vastly different characteristics of SAM with vs. without decaying perturbation size or gradient normalization, and suggest that the intuitions gained from one version may not apply to the other.