A Note on Complexity for Two Classes of Structured Non-Smooth Non-Convex Compositional Optimization

TL;DR

This paper analyzes the complexity of first-order methods for two classes of structured non-smooth, non-convex compositional optimization problems, providing iteration bounds for finding stationary points.

Contribution

It introduces efficient algorithms with proven iteration complexity bounds for structured non-smooth, non-convex compositional problems with specific outer function structures.

Findings

Smoothing compositional gradient method finds a $( abla, abla)$-stationary point in $O(1/( abla abla^2))$ iterations.

Prox-linear method finds an $ abla$-critical point in $O(1/ abla^2)$ iterations.

The methods exploit the special structure of the outer functions to achieve these complexities.

Abstract

This note studies numerical methods for solving compositional optimization problems, where the inner function is smooth, and the outer function is Lipschitz continuous, non-smooth, and non-convex but exhibits one of two special structures that enable the design of efficient first-order methods. In the first structure, the outer function allows for an easily solvable proximal mapping. We demonstrate that, in this case, a smoothing compositional gradient method can find a $(\delta,\epsilon)$-stationary point--specifically defined for compositional optimization--in $O(1/(\delta \epsilon^2))$ iterations. In the second structure, the outer function is expressed as a difference-of-convex function, where each convex component is simple enough to allow an efficiently solvable proximal linear subproblem. In this case, we show that a prox-linear method can find a nearly ${\epsilon}$-critical point in $O(1/\epsilon^2)$ iterations.