Local Saddle Point Optimization: A Curvature Exploitation Approach

TL;DR

This paper introduces a curvature-based optimization method that effectively escapes undesired stationary points in saddle point problems, improving convergence to true local optima in gradient-based methods.

Contribution

The paper presents a novel curvature exploitation approach that enhances existing gradient methods to avoid non-optimal stationary points in saddle problems.

Findings

Curvature exploitation enables escape from undesired stationary points.

Gradient methods with curvature information outperform standard methods.

Empirical results confirm improved convergence in saddle point problems.

Abstract

Gradient-based optimization methods are the most popular choice for finding local optima for classical minimization and saddle point problems. Here, we highlight a systemic issue of gradient dynamics that arise for saddle point problems, namely the presence of undesired stable stationary points that are no local optima. We propose a novel optimization approach that exploits curvature information in order to escape from these undesired stationary points. We prove that different optimization methods, including gradient method and Adagrad, equipped with curvature exploitation can escape non-optimal stationary points. We also provide empirical results on common saddle point problems which confirm the advantage of using curvature exploitation.