Derivative-Free Method For Composite Optimization With Applications To Decentralized Distributed Optimization

TL;DR

This paper introduces a novel derivative-free optimization method combining stochastic zeroth-order and first-order oracles for convex composite problems, with applications to decentralized distributed optimization, achieving near-optimal convergence and communication bounds.

Contribution

It presents the first method using mixed oracles for composite optimization, with proven convergence rates and bounds matching theoretical lower limits in decentralized settings.

Findings

Convergence rate matches first-order methods up to dimension factors.

Communication rounds are optimal, matching lower bounds.

Zeroth-order oracle calls per node are near state-of-the-art.

Abstract

In this paper, we propose a new method based on the Sliding Algorithm from Lan(2016, 2019) for the convex composite optimization problem that includes two terms: smooth one and non-smooth one. Our method uses the stochastic noised zeroth-order oracle for the non-smooth part and the first-order oracle for the smooth part. To the best of our knowledge, this is the first method in the literature that uses such a mixed oracle for the composite optimization. We prove the convergence rate for the new method that matches the corresponding rate for the first-order method up to a factor proportional to the dimension of the space or, in some cases, its squared logarithm. We apply this method for the decentralized distributed optimization and derive upper bounds for the number of communication rounds for this method that matches known lower bounds. Moreover, our bound for the number of zeroth-order oracle calls per node matches the similar state-of-the-art bound for the first-order decentralized distributed optimization up to to the factor proportional to the dimension of the space or, in some cases, even its squared logarithm.