Topological trivialization in non-convex empirical risk minimization

TL;DR

This paper investigates the landscape of non-convex empirical risk functions in high-dimensional settings, establishing conditions under which all local minima are essentially global, leading to a simplified optimization landscape.

Contribution

It introduces a variational formula for the expected number of local minima in non-convex models and identifies conditions for rate trivialization in high-dimensional regimes.

Findings

Expected number of local minima can be controlled asymptotically

Conditions for the vanishing of local minima growth rate are established

Numerical simulations support the conjecture of a phase transition point

Abstract

Given data $\{({\boldsymbol x}_i,y_i): i\le n\}$, with ${\boldsymbol x}_i$ standard $d$-dimensional Gaussian feature vectors, and $y_i\in{\mathbb R}$ response variables, we study the general problem of learning a model parametrized by ${\boldsymbol \theta}\in{\mathbb R}^d$, by minimizing a loss function that depends on ${\boldsymbol \theta}$ via the one-dimensional projections ${\boldsymbol \theta}^{\sf T}{\boldsymbol x}_i$. While previous work mostly dealt with convex losses, our approach assumes general (non-convex) losses hence covering classical, yet poorly understood examples such as the perceptron and non-convex robust regression. We use the Kac-Rice formula to control the asymptotics of the expected number of local minima of the empirical risk, under the proportional asymptotics $n,d\to\infty$, $n/d\to\alpha >1$. Specifically, we prove a finite dimensional variational formula for the exponential growth rate of the expected number of local minima. Further we provide sufficient conditions under which the exponential growth rate vanishes and all empirical risk minimizers have the same asymptotic properties (in fact, we expect the minimizer to be unique in these circumstances). We refer to this phenomenon as `rate trivialization.' If the population risk has a unique minimizer, our sufficient condition for rate trivialization is typically verified when the samples/parameters ratio $\alpha$ is larger than a suitable constant $\alpha_{\star}$. Previous general results of this type required $n\ge Cd \log d$. We illustrate our results in the case of non-convex robust regression. Based on heuristic arguments and numerical simulations, we present a conjecture for the exact location of the trivialization phase transition $\alpha_{\text{tr}}$.