Dimension-Free Bounds for Generalized First-Order Methods via Gaussian Coupling

TL;DR

This paper derives dimension-free, non-asymptotic bounds for generalized first-order algorithms with Gaussian data, using a novel coupling approach that unifies and extends AMP theory without asymptotic assumptions.

Contribution

It introduces a new coupling method that provides tight, finite-sample bounds for a broad class of first-order methods with Gaussian matrices, bypassing classical AMP limitations.

Findings

Establishes explicit Gaussian coupling for iterative algorithms.

Provides tight, dimension-free finite-sample bounds.

Demonstrates the bounds' sharpness with Wasserstein distance lower bounds.

Abstract

We establish non-asymptotic bounds on the finite-sample behavior of generalized first-order iterative algorithms -- including gradient-based optimization methods and approximate message passing (AMP) -- with Gaussian data matrices and full-memory, non-separable nonlinearities. The central result constructs an explicit coupling between the iterates of a generalized first-order method and a conditionally Gaussian process whose covariance evolves deterministically via a finite-dimensional state evolution recursion. This coupling yields tight, dimension-free bounds under mild Lipschitz and moment-matching conditions. Our analysis departs from classical inductive AMP proofs by employing a direct comparison between the generalized first-order method and the conditionally Gaussian comparison process. This approach provides a unified derivation of AMP theory for Gaussian matrices without relying on separability or asymptotics. A complementary lower bound on the Wasserstein distance demonstrates the sharpness of our upper bounds.