Generalization of Zeroth-Order Method for Quotients of Quadratic Functions

TL;DR

This paper introduces a generalized zeroth-order optimization method for quadratic functions and their quotients, using an unconstrained sphere sampling approach, with theoretical analysis and an accelerated algorithm outperforming recent methods.

Contribution

It proposes a novel unconstrained sphere sampling approach for zeroth-order methods, linking to Riemannian optimization and enabling a closed-form solution for step size.

Findings

The proposed method achieves state-of-the-art performance.

The approach effectively estimates Riemannian gradients and Hessians.

The algorithm outperforms recent related methods.

Abstract

Optimization of quadratic functions and the quotient of those are relevant in subspace and iterative optimization methods. In this paper, the calculation of the generalized operator norm and extremal generalized Rayleigh quotient is considered. In contrast to recent works an unconstrained sampling approach on the entire sphere for the random search direction in each iteration is proposed. Furthermore, the link to zeroth-order methods for Riemannian first- and second-order optimization methods is provided in the sense that the Riemannian gradient and Hessian are estimated by the specific surrogates. Even though the tangent space is not used in this construction the optimal step size problem can be computed in a closed form. The subproblems of this and recent works are illuminated in the context of sub-generalized Rayleigh quotient problems on specific Gram matrices. Together the achieved theory allows to construct an accelerated algorithm which shows state-of-the-art behavior and outperforms recent works.