vqSGD: Vector Quantized Stochastic Gradient Descent

TL;DR

This paper introduces vqSGD, a family of vector quantization schemes for distributed optimization that significantly reduce communication costs while maintaining convergence guarantees, leveraging information theory and error-correcting codes.

Contribution

The paper proposes novel vector quantization schemes for stochastic gradient descent that are near optimal, communication-efficient, and provide privacy guarantees.

Findings

Requires o(d) bits for gradient estimation

Achieves asymptotic reduction in communication cost

Provides convergence guarantees and privacy benefits

Abstract

In this work, we present a family of vector quantization schemes \emph{vqSGD} (Vector-Quantized Stochastic Gradient Descent) that provide an asymptotic reduction in the communication cost with convergence guarantees in first-order distributed optimization. In the process we derive the following fundamental information theoretic fact: $\Theta(\frac{d}{R^2})$ bits are necessary and sufficient to describe an unbiased estimator ${\hat{g}}({g})$ for any ${g}$ in the $d$-dimensional unit sphere, under the constraint that $\|{\hat{g}}({g})\|_2\le R$ almost surely. In particular, we consider a randomized scheme based on the convex hull of a point set, that returns an unbiased estimator of a $d$-dimensional gradient vector with almost surely bounded norm. We provide multiple efficient instances of our scheme, that are near optimal, and require only $o(d)$ bits of communication at the expense of tolerable increase in error. The instances of our quantization scheme are obtained using the properties of binary error-correcting codes and provide a smooth tradeoff between the communication and the estimation error of quantization. Furthermore, we show that \emph{vqSGD} also offers strong privacy guarantees.