A Pliable Index Coding Approach to Data Shuffling

TL;DR

This paper explores how pliable index coding can enhance data shuffling efficiency in distributed systems, offering theoretical analysis and a hierarchical scheme that significantly reduces communication costs compared to traditional index coding.

Contribution

It introduces a novel application of pliable index coding for data shuffling, providing theoretical benefits and a hierarchical scheme that outperforms existing index coding methods.

Findings

Achieves up to O(ns/m) reduction in transmissions.

Provides a hierarchical data-shuffling scheme using pliable coding.

Demonstrates theoretical advantages over traditional index coding.

Abstract

A promising research area that has recently emerged, is on how to use index coding to improve the communication efficiency in distributed computing systems, especially for data shuffling in iterative computations. In this paper, we posit that pliable index coding can offer a more efficient framework for data shuffling, as it can better leverage the many possible shuffling choices to reduce the number of transmissions. We theoretically analyze pliable index coding under data shuffling constraints, and design a hierarchical data-shuffling scheme that uses pliable coding as a component. We find benefits up to $O(ns/m)$ over index coding, where $ns/m$ is the average number of workers caching a message, and $m$, $n$, and $s$ are the numbers of messages, workers, and cache size, respectively.