A Smooth Approximation Framework for Weakly Convex Optimization

TL;DR

This paper introduces a unified framework for smooth approximation methods in weakly convex optimization, extending existing techniques and providing complexity guarantees for both deterministic and stochastic problems, including non-Lipschitz functions.

Contribution

It generalizes the concept of smoothable functions, unifies various smoothing techniques, and offers a comprehensive framework with complexity guarantees for weakly convex optimization.

Findings

Unified framework for smoothing techniques in weakly convex optimization.

Complexity guarantees for deterministic and stochastic algorithms.

Extension to non-Lipschitz functions in optimization.

Abstract

Standard complexity analyses for weakly convex optimization rely on the Moreau envelope technique proposed by Davis and Drusvyatskiy (2019). The main insight is that nonsmooth algorithms, such as proximal subgradient, proximal point, and their stochastic variants, implicitly minimize a smooth surrogate function induced by the Moreau envelope. Meanwhile, explicit smoothing, which directly minimizes a smooth approximation of the objective, has long been recognized as an efficient strategy for nonsmooth optimization. In this paper, we generalize the notion of smoothable functions, which was proposed by Beck and Teboulle (2012) for nonsmooth convex optimization. This generalization provides a unified viewpoint on several important smoothing techniques for weakly convex optimization, including Nesterov-type smoothing and Moreau envelope smoothing. Our theory yields a framework for designing smooth approximation algorithms for both deterministic and stochastic weakly convex problems with provable complexity guarantees. Furthermore, our theory extends to the smooth approximation of non-Lipschitz functions, allowing for complexity analysis even when global Lipschitz continuity does not hold.