Trustworthy Efficient Communication for Distributed Learning using LQ-SGD Algorithm

TL;DR

This paper introduces LQ-SGD, a novel gradient compression algorithm for distributed learning that reduces communication costs using low-rank approximation and log-quantization, while maintaining convergence and robustness.

Contribution

LQ-SGD combines low-rank approximation and log-quantization to improve communication efficiency and robustness in distributed training, building on PowerSGD.

Findings

Reduces communication overhead significantly

Maintains training convergence speed and accuracy

Enhances resistance to gradient inversion attacks

Abstract

We propose LQ-SGD (Low-Rank Quantized Stochastic Gradient Descent), an efficient communication gradient compression algorithm designed for distributed training. LQ-SGD further develops on the basis of PowerSGD by incorporating the low-rank approximation and log-quantization techniques, which drastically reduce the communication overhead, while still ensuring the convergence speed of training and model accuracy. In addition, LQ-SGD and other compression-based methods show stronger resistance to gradient inversion than traditional SGD, providing a more robust and efficient optimization path for distributed learning systems.