Asymptotics of Non-Convex Generalized Linear Models in High-Dimensions: A proof of the replica formula

TL;DR

This paper rigorously proves the replica-symmetric formulas for high-dimensional non-convex Generalized Linear Models, confirming physicists' predictions and establishing conditions for their validity using advanced mathematical tools.

Contribution

It introduces a systematic framework combining Gaussian Min-Max Theorem and Approximate Message Passing to validate replica predictions in non-convex high-dimensional optimization.

Findings

Proves the optimality of Tukey loss over Huber loss under contaminated data.

Establishes the optimality of negative regularization in non-convex regression.

Characterizes performance limits of linearized AMP algorithms.

Abstract

The analytic characterization of the high-dimensional behavior of optimization for Generalized Linear Models (GLMs) with Gaussian data has been a central focus in statistics and probability in recent years. While convex cases, such as the LASSO, ridge regression, and logistic regression, have been extensively studied using a variety of techniques, the non-convex case remains far less understood despite its significance. A non-rigorous statistical physics framework has provided remarkable predictions for the behavior of high-dimensional optimization problems, but rigorously establishing their validity for non-convex problems has remained a fundamental challenge. In this work, we address this challenge by developing a systematic framework that rigorously proves replica-symmetric formulas for non-convex GLMs and precisely determines the conditions under which these formulas are valid. Remarkably, the rigorous replica-symmetric predictions align exactly with the conjectures made by physicists, and the so-called replicon condition. The originality of our approach lies in connecting two powerful theoretical tools: the Gaussian Min-Max Theorem, which we use to provide precise lower bounds, and Approximate Message Passing (AMP), which is shown to achieve these bounds algorithmically. We demonstrate the utility of this framework through significant applications: (i) by proving the optimality of the Tukey loss over the more commonly used Huber loss under a $\varepsilon$ contaminated data model, (ii) establishing the optimality of negative regularization in high-dimensional non-convex regression and (iii) characterizing the performance limits of linearized AMP algorithms. By rigorously validating statistical physics predictions in non-convex settings, we aim to open new pathways for analyzing increasingly complex optimization landscapes beyond the convex regime.