Non-smooth stochastic gradient descent using smoothing functions

TL;DR

This paper introduces a smoothing stochastic gradient method for non-smooth compositional optimization, providing convergence guarantees and rates across convex, strongly convex, and non-convex settings.

Contribution

It proposes a novel smoothing-based stochastic gradient approach with proven convergence rates for non-smooth compositional problems in machine learning.

Findings

Achieves a 1/T^(1/4) convergence rate for convex objectives.

Provides convergence guarantees in strongly convex and non-convex settings.

Preliminary results suggest competitiveness of the method on certain problems.

Abstract

In this paper, we address stochastic optimization problems involving a composition of a non-smooth outer function and a smooth inner function, a formulation frequently encountered in machine learning and operations research. To deal with the non-differentiability of the outer function, we approximate the original non-smooth function using smoothing functions, which are continuously differentiable and approach the original function as a smoothing parameter goes to zero (at the price of increasingly higher Lipschitz constants). The proposed smoothing stochastic gradient method iteratively drives the smoothing parameter to zero at a designated rate. We establish convergence guarantees under strongly convex, convex, and non-convex settings, proving convergence rates that match known results for non-smooth stochastic compositional optimization. In particular, for convex objectives, smoothing stochastic gradient achieves a 1/T^(1/4) rate in terms of the number of stochastic gradient evaluations. We further show how general compositional and finite-sum compositional problems (widely used frameworks in large-scale machine learning and risk-averse optimization) fit the assumptions needed for the rates (unbiased gradient estimates, bounded second moments, and accurate smoothing errors). We present preliminary numerical results indicating that smoothing stochastic gradient descent can be competitive for certain classes of problems.