Some Unified Theory for Variance Reduced Prox-Linear Methods

TL;DR

This paper develops a unified convergence theory for variance-reduced prox-linear methods applied to nonconvex, nonsmooth composite optimization problems, improving theoretical guarantees and accommodating inexact computations.

Contribution

It introduces a unified convergence framework that simplifies assumptions, enhances guarantees, and broadens applicability of variance-reduced prox-linear algorithms.

Findings

Operator norm bounds suffice for convergence analysis

State-of-the-art high probability guarantees achieved

Inexact proximal computations are supported

Abstract

This work considers the nonconvex, nonsmooth problem of minimizing a composite objective of the form $f(g(x))+h(x)$ where the inner mapping $g$ is a smooth finite summation or expectation amenable to variance reduction. In such settings, prox-linear methods can enjoy variance-reduced speed-ups despite the existence of nonsmoothness. We provide a unified convergence theory applicable to a wide range of common variance-reduced vector and Jacobian constructions. All the technical conditions we required for variance-reduced methods can be summarized in a single unified assumption. Our theory (i) only requires operator norm bounds on Jacobians (whereas prior works used potentially much larger Frobenius norms), (ii) provides state-of-the-art high probability guarantees, and (iii) allows inexactness in proximal computations.