ZIVR: An Incremental Variance Reduction Technique For Zeroth-Order Composite Problems

TL;DR

This paper introduces ZIVR, a novel incremental variance reduction method for zeroth-order composite optimization that reduces computational costs and achieves convergence rates comparable to first-order methods.

Contribution

We propose ZIVR, a flexible variance reduction framework for zeroth-order optimization that eliminates large batch sampling and provides comprehensive convergence guarantees.

Findings

ZIVR achieves improved convergence rates in various convexity settings.

Numerical experiments demonstrate ZIVR's effectiveness and efficiency.

The method reduces computational costs compared to existing variance reduction techniques.

Abstract

This paper investigates zeroth-order (ZO) finite-sum composite optimization. Recently, variance reduction techniques have been applied to ZO methods to mitigate the non-vanishing variance of 2-point estimators in constrained/composite optimization, yielding improved convergence rates. However, existing ZO variance reduction methods typically involve batch sampling of size at least $\Theta(n)$ or $\Theta(d)$, which can be computationally prohibitive for large-scale problems. In this work, we propose a general variance reduction framework, Zeroth-Order Incremental Variance Reduction (ZIVR), which supports flexible implementations$\unicode{x2014}$including a pure 2-point zeroth-order algorithm that eliminates the need for large batch sampling. Furthermore, we establish comprehensive convergence guarantees for ZIVR across strongly-convex, convex, and non-convex settings that match their first-order counterparts. Numerical experiments validate the effectiveness of our proposed algorithm.