Straggler-Resilient Differentially-Private Decentralized Learning

TL;DR

This paper addresses the challenge of stragglers in decentralized learning while maintaining data privacy, analyzing latency, convergence, and privacy trade-offs through theoretical and empirical methods.

Contribution

It extends differential privacy amplification to include training latency considerations in decentralized learning with stragglers.

Findings

The skipping scheme reduces overall training latency compared to waiting for all nodes.

A trade-off exists between latency, accuracy, and privacy controlled by the timeout parameter.

Empirical validation confirms the theoretical analysis on real-world and benchmark datasets.

Abstract

We consider the straggler problem in decentralized learning over a logical ring while preserving user data privacy. Especially, we extend the recently proposed framework of differential privacy (DP) amplification by decentralization by Cyffers and Bellet to include overall training latency--comprising both computation and communication latency. Analytical results on both the convergence speed and the DP level are derived for both a skipping scheme (which ignores the stragglers after a timeout) and a baseline scheme that waits for each node to finish before the training continues. A trade-off between overall training latency, accuracy, and privacy, parameterized by the timeout of the skipping scheme, is identified and empirically validated for logistic regression on a real-world dataset and for image classification using the MNIST and CIFAR-10 datasets.