State evolution beyond first-order methods I: Rigorous predictions and finite-sample guarantees

TL;DR

This paper introduces a rigorous framework for predicting the behavior of complex iterative algorithms in high-dimensional nonconvex optimization, extending state evolution analysis beyond first-order methods and providing finite-sample guarantees.

Contribution

It develops a general state evolution prediction for algorithms combining first-order and saddle point updates, with finite-sample deviation bounds, advancing analysis tools for nonconvex optimization.

Findings

Established a rigorous state evolution prediction for complex algorithms.

Provided finite-sample guarantees bounding empirical deviations.

Developed a new technical toolkit including Hilbert space lifting and Gaussian comparison techniques.

Abstract

We develop a toolbox for exact analysis of iterative algorithms on a class of high-dimensional nonconvex optimization problems with random data. While prior work has shown that low-dimensional statistics of (generalized) first-order methods can be predicted by a deterministic recursion known as state evolution, our focus is on developing such a prediction for a more general class of algorithms. We provide a state evolution for any method whose iterations are given by (possibly interleaved) first-order and saddle point updates, showing two main results. First, we establish a rigorous state evolution prediction that holds even when the updates are not coordinate-wise separable. Second, we establish finite-sample guarantees bounding the deviation of the empirical updates from the established state evolution. In the process, we develop a technical toolkit that may prove useful in related problems. One component of this toolkit is a general Hilbert space lifting technique to prove existence and uniqueness of a convenient parameterization of the state evolution. Another component of the toolkit combines a generic application of Bolthausen's conditioning method with a sequential variant of Gordon's Gaussian comparison inequality, and provides additional ingredients that enable a general finite-sample analysis.