Riemannian optimization with finite-difference gradient approximations

TL;DR

This paper introduces a new derivative-free Riemannian optimization method using adaptive finite-difference gradients, achieving efficient convergence with fewer function evaluations, and demonstrates superior performance over existing methods.

Contribution

The paper proposes a novel finite-difference gradient approximation technique for Riemannian optimization with adaptive accuracy, improving efficiency and complexity bounds.

Findings

Achieves $O(d\,\epsilon^{-2})$ function evaluations for $\,\epsilon$-critical points.

Provides a variant with extrinsic finite-difference scheme for embedded manifolds.

Numerical results show superior performance over existing derivative-free methods.

Abstract

Derivative-free Riemannian optimization (DFRO) aims to minimize an objective function using only function evaluations, under the constraint that the decision variables lie on a Riemannian manifold. The rapid increase in problem dimensions over the years calls for computationally cheap DFRO algorithms, that is, algorithms requiring as few function evaluations and retractions as possible. We propose a novel DFRO method based on finite-difference gradient approximations that relies on an adaptive selection of the finite-difference accuracy and stepsize that is novel even in the Euclidean setting. When endowed with an intrinsic finite-difference scheme, that measures variations of the objective in tangent directions using retractions, our proposed method requires $O(d\epsilon^{-2})$ function evaluations and retractions to find an $\epsilon$-critical point, where $d$ is the manifold dimension. We then propose a variant of our method when the search space is a Riemannian submanifold of an $n$-dimensional Euclidean space. This variant relies on an extrinsic finite-difference scheme, approximating the Riemannian gradient directly in the embedding space, assuming that the objective function can be evaluated outside of the manifold. This approach leads to worst-case complexity bounds of $O(d\epsilon^{-2})$ function evaluations and $O(\epsilon^{-2})$ retractions. We also present numerical results showing that the proposed methods achieve superior performance over existing derivative-free methods on various problems in both Euclidean and Riemannian settings.