Efficiency of minimizing compositions of convex functions and smooth maps

TL;DR

This paper analyzes the efficiency of algorithms for minimizing sums of convex functions and compositions with smooth maps, introducing methods that improve complexity bounds and include acceleration techniques.

Contribution

It presents a comprehensive analysis of the prox-linear method's efficiency, extending it with smoothing, fast-gradient schemes, and inertial acceleration for composite convex minimization.

Findings

Exact subproblem solutions achieve $ ilde{O}(rac{1}{ ext{epsilon}^2})$ complexity.

Approximate solutions with first-order methods reach $ ilde{O}(rac{1}{ ext{epsilon}^3})$ complexity.

Extension to average of multiple functions with improved complexity bounds.

Abstract

We consider global efficiency of algorithms for minimizing a sum of a convex function and a composition of a Lipschitz convex function with a smooth map. The basic algorithm we rely on is the prox-linear method, which in each iteration solves a regularized subproblem formed by linearizing the smooth map. When the subproblems are solved exactly, the method has efficiency $\mathcal{O}(\varepsilon^{-2})$, akin to gradient descent for smooth minimization. We show that when the subproblems can only be solved by first-order methods, a simple combination of smoothing, the prox-linear method, and a fast-gradient scheme yields an algorithm with complexity $\widetilde{\mathcal{O}}(\varepsilon^{-3})$. The technique readily extends to minimizing an average of $m$ composite functions, with complexity $\widetilde{\mathcal{O}}(m/\varepsilon^{2}+\sqrt{m}/\varepsilon^{3})$ in expectation. We round off the paper with an inertial prox-linear method that automatically accelerates in presence of convexity.