Zeroth-Order Negative Curvature Finding: Escaping Saddle Points without Gradients

TL;DR

This paper introduces two zeroth-order negative curvature finding frameworks that efficiently escape saddle points in nonconvex optimization without gradient or Hessian information, improving query complexity.

Contribution

The paper proposes novel zeroth-order negative curvature finding methods that replace Hessian-vector products, enabling efficient saddle point escape without increasing iteration complexity.

Findings

Frameworks achieve convergence to second-order stationary points

Reduced query complexity compared to prior zeroth-order methods

Applicable to multiple zeroth-order algorithms like ZO-GD, ZO-SGD

Abstract

We consider escaping saddle points of nonconvex problems where only the function evaluations can be accessed. Although a variety of works have been proposed, the majority of them require either second or first-order information, and only a few of them have exploited zeroth-order methods, particularly the technique of negative curvature finding with zeroth-order methods which has been proven to be the most efficient method for escaping saddle points. To fill this gap, in this paper, we propose two zeroth-order negative curvature finding frameworks that can replace Hessian-vector product computations without increasing the iteration complexity. We apply the proposed frameworks to ZO-GD, ZO-SGD, ZO-SCSG, ZO-SPIDER and prove that these ZO algorithms can converge to $(\epsilon,\delta)$-approximate second-order stationary points with less query complexity compared with prior zeroth-order works for finding local minima.