Utilizing Redundancy in Cost Functions for Resilience in Distributed Optimization and Learning

TL;DR

This paper introduces a redundancy-based model to enhance the robustness of distributed optimization and learning algorithms against asynchronous and Byzantine faulty agents, improving their resilience through theoretical and empirical analysis.

Contribution

It proposes a novel $(f, r; \, \\epsilon)$-redundancy model to quantify and leverage redundancy among agents' cost functions for robustness.

Findings

Redundancy modeling improves robustness of DGD and D-SGD.

Theoretical analysis confirms resilience benefits.

Empirical results demonstrate practical effectiveness.

Abstract

This paper considers the problem of resilient distributed optimization and stochastic machine learning in a server-based architecture. The system comprises a server and multiple agents, where each agent has a local cost function. The agents collaborate with the server to find a minimum of their aggregate cost functions. We consider the case when some of the agents may be asynchronous and/or Byzantine faulty. In this case, the classical algorithm of distributed gradient descent (DGD) is rendered ineffective. Our goal is to design techniques improving the efficacy of DGD with asynchrony and Byzantine failures. To do so, we start by proposing a way to model the agents' cost functions by the generic notion of $(f, \,r; \epsilon)$-redundancy where $f$ and $r$ are the parameters of Byzantine failures and asynchrony, respectively, and $\epsilon$ characterizes the closeness between agents' cost functions. This allows us to quantify the level of redundancy present amongst the agents' cost functions, for any given distributed optimization problem. We demonstrate, both theoretically and empirically, the merits of our proposed redundancy model in improving the robustness of DGD against asynchronous and Byzantine agents, and their extensions to distributed stochastic gradient descent (D-SGD) for robust distributed machine learning with asynchronous and Byzantine agents.