From Cursed to Competitive: Closing the ZO-FO Gap via Input-to-State Stability

TL;DR

This paper demonstrates that zeroth-order optimization algorithms can, under certain conditions, achieve convergence rates comparable to first-order methods without extra dimension dependence, by analyzing their stability properties.

Contribution

It introduces a dynamical systems perspective and input-to-state stability analysis to show ZO methods can match FO convergence rates under specific conditions.

Findings

ZO methods do not necessarily suffer from extra dimension dependence in convergence rates.

Under certain conditions, ZO methods converge to a neighborhood of FO fixed points.

Theoretical results are supported by numerical examples.

Abstract

While it is generally understood that zeroth-order (ZO) algorithms have an extra dependency on their number of iterations for any choice of parameters, compared to their first-order (FO) counterparts, in this work, we show that under several conditions, in expectation, ZO methods do not suffer from extra dimension dependencies in their convergence rates with respect to their FO counterparts. We look at optimisation algorithms from the dynamical systems perspective and analyse the conditions under which one can formulate the average of a ZO algorithm as the average of its FO counterpart with bounded perturbations with values dependent on design parameters. Then, using input-to-state stability properties, we show ZO methods follow the same decay rate as their FO counterparts and converge to a neighbourhood of the fixed point of FO methods, where its radius depends on the bound of the norm of the perturbations, which can be made arbitrarily small. The theoretical findings are illustrated via numerical examples.