Locally Linear Convergence for Nonsmooth Convex Optimization via Coupled Smoothing and Momentum

TL;DR

This paper introduces an adaptive accelerated smoothing method for nonsmooth convex optimization, achieving optimal sublinear convergence and local linear rates, validated on various practical problems.

Contribution

It develops a coupled smoothing and momentum technique that extends to sums of nonsmooth functions, with proven convergence guarantees and practical efficiency.

Findings

Global sublinear convergence rate of O(1/k)

Local linear convergence under strong convexity

Observed practical convergence of O(1/k^2) transiently

Abstract

We propose an adaptive accelerated smoothing technique for a nonsmooth convex optimization problem where the smoothing update rule is coupled with the momentum parameter. We also extend the setting to the case where the objective function is the sum of two nonsmooth functions. With regard to convergence rate, we provide the global (optimal) sublinear convergence guarantees of O(1/k), which is known to be provably optimal for the studied class of functions, along with a local linear rate if the nonsmooth term fulfills a so-call locally strong convexity condition. We validate the performance of our algorithm on several problem classes, including regression with the l1-norm (the Lasso problem), sparse semidefinite programming (the MaxCut problem), Nuclear norm minimization with application in model free fault diagnosis, and l_1-regularized model predictive control to showcase the benefits of the coupling. An interesting observation is that although our global convergence result guarantees O(1/k) convergence, we consistently observe a practical transient convergence rate of O(1/k^2), followed by asymptotic linear convergence as anticipated by the theoretical result. This two-phase behavior can also be explained in view of the proposed smoothing rule.