Randomized Polar Codes for Anytime Distributed Machine Learning

TL;DR

This paper introduces a robust distributed computing framework combining randomized sketching and polar codes, enabling efficient approximate and exact linear computations even with slow or unavailable nodes, demonstrated on large-scale tasks.

Contribution

It proposes a novel integration of randomized sketching and polar codes for distributed linear computation with an anytime estimator and low complexity decoding.

Findings

Effective in large-scale matrix multiplication

Scalable to ImageNet-sized computations

Robust to slow or missing nodes

Abstract

We present a novel distributed computing framework that is robust to slow compute nodes, and is capable of both approximate and exact computation of linear operations. The proposed mechanism integrates the concepts of randomized sketching and polar codes in the context of coded computation. We propose a sequential decoding algorithm designed to handle real valued data while maintaining low computational complexity for recovery. Additionally, we provide an anytime estimator that can generate provably accurate estimates even when the set of available node outputs is not decodable. We demonstrate the potential applications of this framework in various contexts, such as large-scale matrix multiplication and black-box optimization. We present the implementation of these methods on a serverless cloud computing system and provide numerical results to demonstrate their scalability in practice, including ImageNet scale computations.