On the Second-order Convergence Properties of Random Search Methods

TL;DR

This paper analyzes the convergence of random search methods for non-convex optimization, showing standard methods converge slowly and proposing a new variant that exploits negative curvature for faster convergence.

Contribution

The paper introduces a novel random search algorithm leveraging negative curvature, achieving linear complexity in problem dimension for second-order stationary points.

Findings

Standard random search converges to second-order stationary points but with exponential complexity.

The proposed method exploits negative curvature, reducing complexity to linear in dimension.

Empirical results confirm the theoretical convergence rates of the new algorithm.

Abstract

We study the theoretical convergence properties of random-search methods when optimizing non-convex objective functions without having access to derivatives. We prove that standard random-search methods that do not rely on second-order information converge to a second-order stationary point. However, they suffer from an exponential complexity in terms of the input dimension of the problem. In order to address this issue, we propose a novel variant of random search that exploits negative curvature by only relying on function evaluations. We prove that this approach converges to a second-order stationary point at a much faster rate than vanilla methods: namely, the complexity in terms of the number of function evaluations is only linear in the problem dimension. We test our algorithm empirically and find good agreements with our theoretical results.