Compressed Distributed Stochastic Nonconvex Optimization with Differential Privacy

TL;DR

This paper introduces a novel compressed distributed stochastic gradient descent algorithm for nonconvex optimization that ensures differential privacy and achieves fast convergence rates, applicable to large-scale multi-agent systems.

Contribution

It proposes a robust algorithm accommodating general compression errors, achieving optimal convergence rates under nonconvex and Polyak--Lojasiewicz conditions, while ensuring differential privacy.

Findings

Achieves first-order stationarity with linear speedup rate

Converges to the global optimum under P--L condition

Ensures differential privacy without losing accuracy

Abstract

This paper studies distributed stochastic nonconvex optimization problems with compressed communication and differential privacy, in which each agent aims to minimize the sum of all agents' cost functions by using local compressed information exchange. To this end, we propose a compressed distributed stochastic gradient descent algorithm, which is robust under a general class of compression operators that allow both relative and absolute compression errors. We then show that the proposed algorithm finds the first-order stationary point for smooth nonconvex functions with the linear speedup convergence rate $\mathcal{O}(1/\sqrt{nT})$ and converges to the optimum if the global cost function additionally satisfies the Polyak--{\L}ojasiewicz (P--\L) condition with the convergence rate $\mathcal{O}(1/(nT^\theta)),\theta\in(0,1)$, where $T$ is the total number of iterations and $n$ is the number of agents. Furthermore, if the P--\L ~constant is known in advance, we show that the proposed algorithm achieves a convergence rate $\mathcal{O}(1/(nT))$. Finally, we show that the proposed algorithm is able to achieve $(0,\delta)$-differential privacy without sacrificing convergence accuracy. Numerical experiments are carried out to