Convergence of Clipped SGD on Convex $(L_0,L_1)$-Smooth Functions

TL;DR

This paper analyzes the convergence of clipped stochastic gradient descent on convex functions with a generalized smoothness condition, providing theoretical guarantees and empirical validation for the proposed methods.

Contribution

It introduces convergence analysis for clipped SGD under $(L_0,L_1)$-smoothness and proposes an adaptive variant with matching guarantees.

Findings

High probability convergence rate similar to standard SGD

Effective adaptive SGD variant with gradient clipping

Empirical results support theoretical claims

Abstract

We study stochastic gradient descent (SGD) with gradient clipping on convex functions under a generalized smoothness assumption called $(L_0,L_1)$-smoothness. Using gradient clipping, we establish a high probability convergence rate that matches the SGD rate in the $L$ smooth case up to polylogarithmic factors and additive terms. We also propose a variation of adaptive SGD with gradient clipping, which achieves the same guarantee. We perform empirical experiments to examine our theory and algorithmic choices.