Fundamental Resource Trade-offs for Encoded Distributed Optimization

TL;DR

This paper advances the understanding of encoded optimization in distributed systems, analyzing how data redundancy and straggler tolerance affect convergence, accuracy, and computational efficiency in large-scale data analysis.

Contribution

It provides new mathematical insights into encoded optimization, broadening its applicability and characterizing fundamental trade-offs in distributed iterative algorithms.

Findings

Demonstrates effectiveness of encoded optimization in diverse settings.

Analyzes convergence behavior of iterative algorithms under data redundancy.

Characterizes trade-offs between convergence rate, data size, accuracy, and straggler tolerance.

Abstract

Dealing with the shear size and complexity of today's massive data sets requires computational platforms that can analyze data in a parallelized and distributed fashion. A major bottleneck that arises in such modern distributed computing environments is that some of the worker nodes may run slow. These nodes a.k.a.~stragglers can significantly slow down computation as the slowest node may dictate the overall computational time. A recent computational framework, called encoded optimization, creates redundancy in the data to mitigate the effect of stragglers. In this paper we develop novel mathematical understanding for this framework demonstrating its effectiveness in much broader settings than was previously understood. We also analyze the convergence behavior of iterative encoded optimization algorithms, allowing us to characterize fundamental trade-offs between convergence rate, size of data set, accuracy, computational load (or data redundancy), and straggler toleration in this framework.