New Aspects of Black Box Conditional Gradient: Variance Reduction and One Point Feedback

TL;DR

This paper introduces JAGUAR, a new zero-order gradient approximation method for black-box optimization that reduces variance, requires minimal oracle calls, and is effective in both deterministic and stochastic settings, with proven convergence.

Contribution

The paper proposes JAGUAR, a novel variance-reducing gradient estimator for black-box optimization, applicable to Frank-Wolfe and Gradient Descent algorithms, with theoretical convergence guarantees.

Findings

JAGUAR outperforms existing estimators in experiments.

The method is robust in stochastic and deterministic scenarios.

Convergence is proven for non-convex, convex, and PL-condition cases.

Abstract

This paper deals with the black-box optimization problem. In this setup, we do not have access to the gradient of the objective function, therefore, we need to estimate it somehow. We propose a new type of approximation JAGUAR, that memorizes information from previous iterations and requires $\mathcal{O}(1)$ oracle calls. We implement this approximation in the Frank-Wolfe and Gradient Descent algorithms and prove the convergence of these methods with different types of zero-order oracle. Our theoretical analysis covers scenarios of non-convex, convex and PL-condition cases. Also in this paper, we consider the stochastic minimization problem on the set $Q$ with noise in the zero-order oracle; this setup is quite unpopular in the literature, but we prove that the JAGUAR approximation is robust not only in deterministic minimization problems, but also in the stochastic case. We perform experiments to compare our gradient estimator with those already known in the literature and confirm the dominance of our methods.