Trustworthiness of Stochastic Gradient Descent in Distributed Learning

TL;DR

This paper evaluates the trustworthiness of compressed SGD in distributed learning, showing it offers better privacy resistance than uncompressed SGD and questioning the reliability of MIA as a privacy metric.

Contribution

It provides the first empirical assessment of trustworthiness in compressed SGD, highlighting its enhanced privacy resistance and challenging existing privacy evaluation metrics.

Findings

Compressed SGD shows higher resistance to gradient inversion attacks.

Membership inference attacks may not reliably measure privacy risks.

Compressed SGD can improve scalability without compromising trustworthiness.

Abstract

Distributed learning (DL) uses multiple nodes to accelerate training, enabling efficient optimization of large-scale models. Stochastic Gradient Descent (SGD), a key optimization algorithm, plays a central role in this process. However, communication bottlenecks often limit scalability and efficiency, leading to increasing adoption of compressed SGD techniques to alleviate these challenges. Despite addressing communication overheads, compressed SGD introduces trustworthiness concerns, as gradient exchanges among nodes are vulnerable to attacks like gradient inversion (GradInv) and membership inference attacks (MIA). The trustworthiness of compressed SGD remains unexplored, leaving important questions about its reliability unanswered. In this paper, we provide a trustworthiness evaluation of compressed versus uncompressed SGD. Specifically, we conducted empirical studies using GradInv attacks, revealing that compressed SGD demonstrates significantly higher resistance to privacy leakage compared to uncompressed SGD. In addition, our findings suggest that MIA may not be a reliable metric for assessing privacy risks in distributed learning.