Complexity and convergence analysis of a single-loop SDCAM for Lipschitz composite optimization and beyond

TL;DR

This paper introduces a novel single-loop algorithm for complex composite optimization problems involving Lipschitz and non-Lipschitz functions, providing convergence guarantees and complexity analysis.

Contribution

It extends the SDCAM algorithm to handle more general non-Lipschitz functions h, with proven iteration complexity and convergence properties.

Findings

Matches best known complexity for Lipschitz h

Achieves $ ilde{O}( ext{epsilon}^{-4})$ complexity for continuous h with compact domain

Ensures accumulation points satisfy stationarity conditions

Abstract

We develop and analyze a single-loop algorithm for minimizing the sum of a Lipschitz differentiable function $f$, a prox-friendly proper closed function $g$ (with a closed domain on which $g$ is continuous) and the composition of another prox-friendly proper closed function $h$ (whose domain is closed on which $h$ is continuous) with a continuously differentiable mapping $c$ (that is Lipschitz continuous and Lipschitz differentiable on the convex closure of the domain of $g$). Such models arise naturally in many contemporary applications, where $f$ is the loss function for data misfit, and $g$ and $h$ are nonsmooth functions for inducing desirable structures in $x$ and $c(x)$. Existing single-loop algorithms mainly focus either on the case where $h$ is Lipschitz continuous or the case where $h$ is an indicator function of a closed convex set. In this paper, we develop a single-loop algorithm for more general possibly non-Lipschitz $h$. Our algorithm is a single-loop variant of the successive difference-of-convex approximation method (SDCAM) proposed in [22]. We show that when $h$ is Lipschitz, our algorithm exhibits an iteration complexity that matches the best known complexity result for obtaining an $(\epsilon_1,\epsilon_2,0)$-stationary point. Moreover, we show that, by assuming additionally that dom $g$ is compact, our algorithm exhibits an iteration complexity of $\tilde{O}(\epsilon^{-4})$ for obtaining an $(\epsilon,\epsilon,\epsilon)$-stationary point when $h$ is merely continuous and real-valued. Furthermore, we consider a scenario where $h$ does not have full domain and establish vanishing bounds on successive changes of iterates. Finally, in all three cases mentioned above, we show that one can construct a subsequence such that any accumulation point $x^*$ satisfies $c(x^*)\in$ dom $h$, and if a standard constraint qualification holds at $x^*$, then $x^*$ is a stationary point.