Compression for Distributed Optimization and Timely Updates

TL;DR

This thesis systematically investigates compression techniques in distributed optimization, establishing theoretical bounds, developing efficient quantizers, and designing schemes for timely data transmission to enhance distributed computing performance.

Contribution

It introduces new bounds, quantizers, and algorithms for gradient compression, mean estimation, and entropic compression with a focus on timely updates in distributed systems.

Findings

Established information theoretic lower bounds for gradient precision.

Developed fast quantizers matching theoretical bounds.

Designed efficient schemes for timely entropic compression.

Abstract

The goal of this thesis is to study the compression problems arising in distributed computing systematically. In the first part of the thesis, we study gradient compression for distributed first-order optimization. We begin by establishing information theoretic lower bounds on optimization accuracy when only finite precision gradients are used. Also, we develop fast quantizers for gradient compression, which, when used with standard first-order optimization algorithms, match the aforementioned lower bounds. In the second part of the thesis, we study distributed mean estimation, an important primitive for distributed optimization algorithms. We develop efficient estimators which improve over state of the art by efficiently using the side information present at the center. We also revisit the Gaussian rate-distortion problem and develop efficient quantizers for this problem in both the side-information and the no side information setting. Finally, we study the problem of entropic compression of the symbols transmitted by the edge devices to the center, which commonly arise in cyber-physical systems. Our goal is to design entropic compression schemes that allow the information to be transmitted in a 'timely' manner, which, in turn, enables the center to have access to the latest information for computation. We shed light on the structure of the optimal entropic compression scheme and, using this structure, we develop efficient algorithms to compute this optimal compression scheme.