Zeroth-Order Stochastic Coordinate Methods for Decentralized Non-convex Optimization

TL;DR

This paper introduces ZOOM, a zeroth-order coordinate method for decentralized non-convex optimization, enhanced with a 'powerball' mechanism, demonstrating faster convergence in black-box tasks compared to existing algorithms.

Contribution

The paper proposes ZOOM and ZOOM-PB algorithms for decentralized zeroth-order optimization, incorporating a novel acceleration mechanism and validating improved convergence over prior methods.

Findings

Faster convergence demonstrated in benchmark tasks.

Effective acceleration with the 'powerball' mechanism.

Comparable or superior performance to state-of-the-art algorithms.

Abstract

In this letter, we first propose a \underline{Z}eroth-\underline{O}rder c\underline{O}ordinate \underline{M}ethod~(ZOOM) to solve the stochastic optimization problem over a decentralized network with only zeroth-order~(ZO) oracle feedback available. Moreover, we equip a simple mechanism "powerball" to ZOOM and propose ZOOM-PB to accelerate the convergence of ZOOM. Compared with the existing methods, we verify the proposed algorithms through two benchmark examples in the literature, namely the black-box binary classification and the generating adversarial examples from black-box DNNs in order to compare with the existing state-of-the-art centralized and distributed ZO algorithms. The numerical results demonstrate a faster convergence rate of the proposed algorithms.