Progressive Power Homotopy for Non-convex Optimization

TL;DR

This paper introduces Prog-PowerHP, a novel first-order method for non-convex optimization that progressively transforms the objective to improve convergence to near-global optima, especially in complex landscapes.

Contribution

The paper presents a new optimization algorithm that combines power transformation and Gaussian smoothing with progressive parameter updates, providing convergence guarantees and empirical advantages.

Findings

Converges to a neighborhood of the global optimum with nearly quadratic complexity in dimension.

Outperforms standard methods in phase retrieval near the information-theoretic limit.

Effective in training under-parameterized neural networks.

Abstract

We propose a novel first-order method for non-convex optimization of the form $\max_{\bm{w}\in\mathbb{R}^d}\mathbb{E}_{\bm{x}\sim\mathcal{D}}[f_{\bm{w}}(\bm{x})]$, termed Progressive Power Homotopy (Prog-PowerHP). The method applies stochastic gradient ascent to a surrogate objective obtained by first performing a power transformation and then Gaussian smoothing, $F_{N,\sigma}(\bm{\mu}):=\mathbb{E}_{\bm{w}\sim\mathcal{N}(\bm{\mu},\sigma^2I_d),\bm{x}\sim\mathcal{D}}[e^{Nf_w(\bm{x})}]$, while progressively increasing the power parameter $N$ and decreasing the smoothing scale $\sigma$ along the optimization trajectory. We prove that, under mild regularity conditions, Prog-PowerHP converges to a small neighborhood of the global optimum with an iteration complexity scaling nearly as $O(d^2\varepsilon^{-2})$. Empirically, Prog-PowerHP demonstrates clear advantages in phase retrieval when the samples-to-dimension ratio approaches the information-theoretic limit, and in training two-layer neural networks in under-parameterized regimes. These results suggest that Prog-PowerHP is particularly effective for navigating cluttered non-convex landscapes where standard first-order methods struggle.