Hierarchical Gradient Coding: From Optimal Design to Privacy at Intermediate Nodes

TL;DR

This paper develops a hierarchical gradient coding scheme that optimizes communication and computation trade-offs, incorporates privacy at relays, and handles straggling and adversarial nodes in distributed gradient computation.

Contribution

It introduces a linear coding scheme for hierarchical gradient coding that achieves optimal communication loads and ensures privacy without extra communication overhead.

Findings

Achieves optimal relay-to-server and worker-to-relay communication loads.

Provides a privacy-preserving coding scheme using shared randomness.

Handles straggling and adversarial nodes effectively.

Abstract

Gradient coding is a distributed computing technique for computing gradient vectors over large datasets by outsourcing partial computations to multiple workers, typically connected directly to the server. In this work, we investigate gradient coding in a hierarchical setting, where intermediate nodes sit between the server and workers. This structure reduces the communication load received at the server, which is a bottleneck in conventional gradient coding systems. In this paper, the intermediate nodes, referred to as \textit{relays}, process the data received from workers and send the results to the server for the final gradient computation. Our main contribution is deriving the optimal communication-computation trade-off by designing a linear coding scheme, also considering straggling and adversarial nodes among both relays and workers. We propose a coding scheme which achieves both the optimal relay-to-server communication load and the optimal worker-to-relay communication load. We further extend our setting to incorporate privacy by requiring that relays learn no information about the computed partial gradients from the messages they receive. This is achieved by introducing shared randomness among workers, allowing each worker to encode its partial gradients such that the randomness cannot be canceled out at the relay. Meanwhile, the server can successfully decode the global gradient by eliminating this randomness after receiving the computations of the non-straggling relays. Importantly, this privacy guarantee is achieved without increasing the overall communication load.