Double Quantization for Communication-Efficient Distributed Optimization

TL;DR

This paper introduces double quantization, a method to reduce communication costs in distributed machine learning by quantizing both model parameters and gradients, with algorithms that maintain performance while saving bits.

Contribution

It proposes a general double quantization scheme and three novel algorithms, including asynchronous, sparsified, and accelerated variants, with theoretical guarantees and practical validation.

Findings

Significant reduction in communication bits without performance loss.

Effective integration of gradient sparsification with double quantization.

Algorithms demonstrate practical efficiency on multi-server setups.

Abstract

Modern distributed training of machine learning models suffers from high communication overhead for synchronizing stochastic gradients and model parameters. In this paper, to reduce the communication complexity, we propose \emph{double quantization}, a general scheme for quantizing both model parameters and gradients. Three communication-efficient algorithms are proposed under this general scheme. Specifically, (i) we propose a low-precision algorithm AsyLPG with asynchronous parallelism, (ii) we explore integrating gradient sparsification with double quantization and develop Sparse-AsyLPG, (iii) we show that double quantization can also be accelerated by momentum technique and design accelerated AsyLPG. We establish rigorous performance guarantees for the algorithms, and conduct experiments on a multi-server test-bed to demonstrate that our algorithms can effectively save transmitted bits without performance degradation.