Collaborative Compressors in Distributed Mean Estimation with Limited Communication Budget

TL;DR

This paper introduces four simple, computationally efficient collaborative compression schemes for distributed mean estimation that leverage vector similarities to reduce communication costs, with theoretical analysis of error degradation as dissimilarity increases.

Contribution

The paper proposes four novel correlation-agnostic compression schemes that exploit vector similarities in distributed settings, improving communication efficiency without prior correlation knowledge.

Findings

Significant reduction in communication costs achieved.

Error metrics degrade gracefully with increasing dissimilarity.

Schemes are simple to implement and computationally efficient.

Abstract

Distributed high dimensional mean estimation is a common aggregation routine used often in distributed optimization methods. Most of these applications call for a communication-constrained setting where vectors, whose mean is to be estimated, have to be compressed before sharing. One could independently encode and decode these to achieve compression, but that overlooks the fact that these vectors are often close to each other. To exploit these similarities, recently Suresh et al., 2022, Jhunjhunwala et al., 2021, Jiang et al, 2023, proposed multiple correlation-aware compression schemes. However, in most cases, the correlations have to be known for these schemes to work. Moreover, a theoretical analysis of graceful degradation of these correlation-aware compression schemes with increasing dissimilarity is limited to only the $\ell_2$-error in the literature. In this paper, we propose four different collaborative compression schemes that agnostically exploit the similarities among vectors in a distributed setting. Our schemes are all simple to implement and computationally efficient, while resulting in big savings in communication. The analysis of our proposed schemes show how the $\ell_2$, $\ell_\infty$ and cosine estimation error varies with the degree of similarity among vectors.