Distributed storage algorithms with optimal tradeoffs

TL;DR

This paper introduces algorithms that achieve the optimal tradeoff between network traffic and storage overhead in distributed storage systems, ensuring reliable long-term data storage despite node failures.

Contribution

The paper presents algorithms that asymptotically attain the fundamental capacity bound, establishing the optimal tradeoff between repair rate and storage overhead.

Findings

Algorithms achieve the theoretical capacity bound asymptotically.

Optimal tradeoff between network traffic and storage overhead demonstrated.

Provides a fundamental limit for distributed storage reliability.

Abstract

One of the primary objectives of a distributed storage system is to reliably store large amounts of source data for long durations using a large number $N$ of unreliable storage nodes, each with $c$ bits of storage capacity. Storage nodes fail randomly over time and are replaced with nodes of equal capacity initialized to zeroes, and thus bits are erased at some rate $e$. To maintain recoverability of the source data, a repairer continually reads data over a network from nodes at an average rate $r$, and generates and writes data to nodes based on the read data. The distributed storage source capacity is the maximum amount of source that can be reliably stored for long periods of time. Previous research shows that asymptotically the distributed storage source capacity is at most $\left(1-\frac{e}{2 \cdot r}\right) \cdot N \cdot c$ as $N$ and $r$ grow. In this work we introduce and analyze algorithms such that asymptotically the distributed storage source data capacity is at least the above equation. Thus, the above equation expresses a fundamental trade-off between network traffic and storage overhead to reliably store source data.