Subgradient Regularization: A Descent-Oriented Subgradient Method for Nonsmooth Optimization

TL;DR

This paper introduces a unifying descent framework for nonsmooth optimization that uses subgradient regularization to generate stable descent directions, ensuring convergence to stationary points.

Contribution

It develops a general framework for descent methods in nonsmooth optimization and introduces subgradient regularization, providing a new perspective and convergence guarantees.

Findings

Framework provably converges to stationary points.

Effective in optimizing nonsmooth functions like Chebyshev-Rosenbrock.

Recovers and interprets classical prox-linear method.

Abstract

In nonsmooth optimization, a negative subgradient is not necessarily a descent direction, making the design of convergent descent methods based on zeroth-order and first-order information a challenging task. The well-studied bundle methods and gradient sampling algorithms construct descent directions by aggregating subgradients at nearby points in seemingly different ways, and are often complicated or lack deterministic guarantees. In this work, we identify a unifying principle behind these approaches, and develop a general framework of descent methods under the abstract principle that provably converge to stationary points. Within this framework, we introduce a simple yet effective technique, called subgradient regularization, to generate stable descent directions for a broad class of nonsmooth marginal functions, including finite maxima or minima of smooth functions. When applied to the composition of a convex function with a smooth map, the method naturally recovers the prox-linear method and, as a byproduct, provides a new dual interpretation of this classical algorithm. Numerical experiments demonstrate the effectiveness of our methods on several challenging classes of nonsmooth optimization problems, including the minimization of Nesterov's nonsmooth Chebyshev-Rosenbrock function.