Single Loop Gaussian Homotopy Method for Non-convex Optimization

TL;DR

This paper introduces a single loop Gaussian homotopy (SLGH) method for non-convex optimization that reduces computational costs and converges faster than existing methods by updating parameters and variables simultaneously.

Contribution

The paper proposes a novel single loop framework for Gaussian homotopy methods that improves efficiency and convergence in non-convex optimization.

Findings

SLGH converges faster than double loop GH methods.

SLGH outperforms gradient descent in solution quality.

Theoretical analysis shows SLGH's convergence rate matches gradient descent.

Abstract

The Gaussian homotopy (GH) method is a popular approach to finding better stationary points for non-convex optimization problems by gradually reducing a parameter value $t$, which changes the problem to be solved from an almost convex one to the original target one. Existing GH-based methods repeatedly call an iterative optimization solver to find a stationary point every time $t$ is updated, which incurs high computational costs. We propose a novel single loop framework for GH methods (SLGH) that updates the parameter $t$ and the optimization decision variables at the same. Computational complexity analysis is performed on the SLGH algorithm under various situations: either a gradient or gradient-free oracle of a GH function can be obtained for both deterministic and stochastic settings. The convergence rate of SLGH with a tuned hyperparameter becomes consistent with the convergence rate of gradient descent, even though the problem to be solved is gradually changed due to $t$. In numerical experiments, our SLGH algorithms show faster convergence than an existing double loop GH method while outperforming gradient descent-based methods in terms of finding a better solution.