Fundamental Limits of Decentralized Data Shuffling

TL;DR

This paper investigates the fundamental limits of decentralized data shuffling in distributed machine learning, proposing bounds and schemes that optimize communication efficiency under storage constraints and multiple workers.

Contribution

It introduces novel converse bounds and achievable schemes for decentralized data shuffling with asymmetric storage, achieving near-optimal performance under specific conditions.

Findings

Bounds within 3/2 factor of each other for uncoded storage

Schemes are optimal for large storage or up to four workers

Decentralized approach reduces communication load compared to master-worker models

Abstract

Data shuffling of training data among different computing nodes (workers) has been identified as a core element to improve the statistical performance of modern large-scale machine learning algorithms. Data shuffling is often considered as one of the most significant bottlenecks in such systems due to the heavy communication load. Under a master-worker architecture (where a master has access to the entire dataset and only communication between the master and the workers is allowed) coding has been recently proved to considerably reduce the communication load. This work considers a different communication paradigm referred to as decentralized data shuffling, where workers are allowed to communicate with one another via a shared link. The decentralized data shuffling problem has two phases: workers communicate with each other during the data shuffling phase, and then workers update their stored content during the storage phase. The main challenge is to derive novel converse bounds and achievable schemes for decentralized data shuffling by considering the asymmetry of the workers' storages (i.e., workers are constrained to store different files in their storages based on the problem setting), in order to characterize the fundamental limits of this problem. For the case of uncoded storage (i.e., each worker directly stores a subset of bits of the dataset), this paper proposes converse and achievable bounds (based on distributed interference alignment and distributed clique-covering strategies) that are within a factor of 3/2 of one another. The proposed schemes are also exactly optimal under the constraint of uncoded storage for either large storage size or at most four workers in the system.