Power Homotopy for Zeroth-Order Non-Convex Optimizations

TL;DR

This paper introduces GS-PowerHP, a new zeroth-order optimization method for non-convex problems that uses a power-transformed Gaussian surrogate and decaying smoothing parameter, achieving strong theoretical guarantees and empirical performance.

Contribution

The paper presents GS-PowerHP, a novel zeroth-order method combining power-transformed Gaussian smoothing with decaying variance, providing convergence guarantees and state-of-the-art empirical results.

Findings

Converges to a neighborhood of the global maximizer with $O(d^2 \\varepsilon^{-2})$ complexity.

Ranks among top three algorithms on benchmark tests.

Achieves first place in high-dimensional black-box attack experiments.

Abstract

We introduce GS-PowerHP, a novel zeroth-order method for non-convex optimization problems of the form $\max_{x \in \mathbb{R}^d} f(x)$. Our approach leverages two key components: a power-transformed Gaussian-smoothed surrogate $F_{N,\sigma}(\mu) = \mathbb{E}_{x\sim\mathcal{N}(\mu,\sigma^2 I_d)}[e^{N f(x)}]$ whose stationary points cluster near the global maximizer $x^*$ of $f$ for sufficiently large $N$, and an incrementally decaying $\sigma$ for enhanced data efficiency. Under mild assumptions, we prove convergence in expectation to a small neighborhood of $x^*$ with the iteration complexity of $O(d^2 \varepsilon^{-2})$. Empirical results show our approach consistently ranks among the top three across a suite of competing algorithms. Its robustness is underscored by the final experiment on a substantially high-dimensional problem ($d=150,528$), where it achieved first place on least-likely targeted black-box attacks against images from ImageNet, surpassing all competing methods.