3LC: Lightweight and Effective Traffic Compression for Distributed Machine Learning

TL;DR

3LC is a novel traffic compression scheme for distributed machine learning that significantly reduces communication overhead while maintaining model accuracy, leading to substantial training time improvements.

Contribution

It introduces a new lossy compression method combining quantization, sparsification, and encoding techniques, achieving high compression ratios with minimal accuracy loss and no need for modifying existing ML frameworks.

Findings

Achieves up to 107X data compression ratio.

Reduces training time of ResNet-110 on CIFAR-10 by up to 23X.

Maintains nearly the same test accuracy as uncompressed training.

Abstract

The performance and efficiency of distributed machine learning (ML) depends significantly on how long it takes for nodes to exchange state changes. Overly-aggressive attempts to reduce communication often sacrifice final model accuracy and necessitate additional ML techniques to compensate for this loss, limiting their generality. Some attempts to reduce communication incur high computation overhead, which makes their performance benefits visible only over slow networks. We present 3LC, a lossy compression scheme for state change traffic that strikes balance between multiple goals: traffic reduction, accuracy, computation overhead, and generality. It combines three new techniques---3-value quantization with sparsity multiplication, quartic encoding, and zero-run encoding---to leverage strengths of quantization and sparsification techniques and avoid their drawbacks. It achieves a data compression ratio of up to 39--107X, almost the same test accuracy of trained models, and high compression speed. Distributed ML frameworks can employ 3LC without modifications to existing ML algorithms. Our experiments show that 3LC reduces wall-clock training time of ResNet-110--based image classifiers for CIFAR-10 on a 10-GPU cluster by up to 16--23X compared to TensorFlow's baseline design.